Get the App
SLTechnology News&Howtos  ›  Servers  › 

Summary of common commands for statistical analysis of Nginx logs

Shulou Source: shulou.com Published: 2022-06-02 02:14:58 11月18日 Update

This article mainly summarizes some common commands about the statistical analysis of Nginx logs and shares them for your reference and study. Let's take a look at the detailed introduction:

I. related statistics of IP

Statistics of IP visits

Awk'{print $1} 'access.log | sort-n | uniq | wc-l

Check the number of IP visits in a certain period of time (4-5 o'clock)

Grep "07/Apr/2017:0 [4-5]" access.log | awk'{print $1}'| sort | uniq-c | sort-nr | wc-l

Check out the top 100 most frequently visited IP

Awk'{print $1} 'access.log | sort-n | uniq-c | sort-rn | head-n 100

View IP that have been visited more than 100 times

Awk'{print $1} 'access.log | sort-n | uniq-c | awk' {if ($1 > 100) print $0}'| sort-rn

Query the detailed access of an IP, sorted by access frequency

Grep '104.217.108.66' access.log | awk' {print $7}'| sort | uniq-c | sort-rn | head-n 100

Second, page visit statistics

View the most visited page (TOP100)

Awk'{print $7} 'access.log | sort | uniq-c | sort-rn | head-n 100

View the most frequently visited pages ([exclude php pages] (TOP100)

Grep-v ".php" access.log | awk'{print $7}'| sort | uniq-c | sort-rn | head-n 100

View pages that have been visited more than 100 times

Cat access.log | cut-d''- f 7 | sort | uniq-c | awk'{if ($1 > 100) print $0}'| less

View the most visited page with the last 1000 records

Tail-1000 access.log | awk'{print $7}'| sort | uniq-c | sort-nr | less

Statistics of requests per second

Count the number of requests per second and the time point of top100 (accurate to seconds)

Awk'{print $4} 'access.log | cut-c 14-21 | sort | uniq-c | sort-nr | head-n 100

Statistics of requests per minute

Count the number of requests per minute and the time point of top100 (accurate to minutes)

Awk'{print $4} 'access.log | cut-c 14-18 | sort | uniq-c | sort-nr | head-n 100

Hourly request statistics

Count the number of requests per hour and the time point of top100 (accurate to hours)

Awk'{print $4} 'access.log | cut-c 14-15 | sort | uniq-c | sort-nr | head-n 100

Third, performance analysis

Add $request_time to the last field in nginx log

List the pages with a transmission time of more than 3 seconds, and display the first 20

Cat access.log | awk'($NF > 3) {print $7}'| sort-n | uniq-c | sort-nr | head-20

List the pages with a request time of more than 3 seconds for php pages, and count the number of times they appear, showing the first 100 entries

Cat access.log | awk'($NF > 1 & & $7 dollars /\ .php /) {print $7}'| sort-n | uniq-c | sort-nr | head-100

IV. Spider capture statistics

Statistics on the number of spiders crawling

Grep 'Baiduspider' access.log | wc-l

Count the number of spiders crawling 404

Grep 'Baiduspider' access.log | grep' 404' | wc-l

Fifth, TCP connection statistics

View current TCP connections

Netstat-tan | grep "ESTABLISHED" | grep ": 80" | wc-l

Use tcpdump to sniff the access to port 80 to see who is the highest.

Tcpdump-I eth0-tnn dst port 80-c 1000 | awk-F "."'{print $1 "." $2 "." $3 "." $4}'| sort | uniq-c | sort-nr

VI. Example script

Get the number of nginx access log entries in the last minute

#! / bin/bash export LANG=Cexport PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binTIME=$ (date-d "1 minute ago" + "% d/%h/%Y:%H:%M") grep "$TIME" / var/log/nginx/access.log | wc-l

Get the number of nginx error log entries in the previous minute

#! / bin/bash export LANG=Cexport PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binTIME=$ (date-d "1 minute ago" + "% Y-%m-%d% HRV% M") grep "$TIME" / var/log/nginx/error.log | wc-l

Summary

The above is the whole content of this article, I hope that the content of this article can bring some help to your study or work, if you have any questions, you can leave a message and exchange, thank you for your support.

Tags: Statistics pages time times logs precision hours spiders visits analysis maximum content learning commands common statistical analysis frequency fields examples that is Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno vpn Apple Shulou Information Microsoft MySQL