Slow log cutting of multi-project hundred libraries and hundred meters
Problem description:
Multiple projects, some projects sub-library sub-table distributed in multiple nodes, therefore, the cutting and collection of slow logs is extremely necessary for slow log analysis.
Implementation plan:
① environment preparation
Host1: ftp server (192.168.xx.xxx) for centralized storage of daily (scriptable) logs for each project
Host2: mysql server (192.168.xx.xxx), which stores configuration-related parameters (such as log host,user,pwd, log location, and so on) that are then used for query invocation of scripts.
Host3: the machine that performs the operation (192.168.xx.xxx), that is, the machine that executes the scheduled tasks set by the script
② script preparation
Involves two scripts, logrotate.sh and job.sh
Where job.sh is used for batch execution of related host operations stored in mysql, and job.sh calls logrotate.sh
Logrotate.sh is used to actually perform operations on each machine that needs to collect logs.
The structure and data of the mysql configuration table are as follows (please add an account where all machines can query the table):
The job.sh script is as follows:
Click (here) to collapse or open
#! / bin/bash
Mysql_host=192.168.xx.xxx
Mysql_port=3456
Mysql_user=xxx
Mysql_pwd=xxxxxx
Function get_info {
Cmd= `which mysql`
$cmd-h$mysql_host-P$mysql_port-u$mysql_user-p$mysql_pwd-BNe "select * from DB_SPLIT_LOG.t_log_info;" | while read line
Do
Host_ip= `echo $line | awk'{print $3}'`
Ssh_user= `echo $line | awk'{print $9}'`
Ssh_pwd= `echo $line | awk'{print $10}'`
/ usr/bin/sshpass-p $ssh_pwd ssh-o StrictHostKeyChecking=no $ssh_user@$host_ip "bash" $tran_log
Exec 2 > $tran_log
}
Function create_rotate_conf {
Echo ""
$slow_log {
Daily
Copytruncate
Dateext
Create 0660 mysql mysql
Olddir $old_dir
Rotate 1
}
"" > $tmp_conf
}
Function start_logrotate {
Echo ""
/ usr/sbin/logrotate-vf $tmp_conf
}
Function change_logname {
Echo ""
For file in `find $old_dir-maxdepth 1-type f-name "* slow*$ {date} *" `
Do
Echo $file
Done
Echo "change log name $file to ${old_dir} / ${log_name}"
Mv $file ${old_dir} / ${log_name}
}
Function ftp_put {
Echo ""
Echo "start ftp put $log_name"
Ftp-in
