Logstash inserts the parsed log into the MySQL database
1. First of all, Logstash needs to install the logstash-output-jdbc plug-in (the following is based on docker)
Dockerfile is as follows:
FROM logstash:7.4.0RUN sed-I's # https://rubygems.org#https://gems.ruby-china.com#g' Gemfile & & logstash-plugin install logstash-output-jdbc
two。 Parsing nginx Log
Date {match = > ["timestamp" "dd/MMM/yyyy:HH:mm:ss Z"] # parsing time} urldecode {field = > "Http_Request" # request log to Chinese} geoip {# to get IP plug-in database = > "/ usr/share/logstash/geodb/GeoLite2-City.mmdb" source = > "Client_" IP "target = >" geoip "fields = > [" country_name "" "region_name", "city_name", "ip", "longitude", "latitude", "location"] add_field = > ["[geoip] [coordinates]", "% {[geoip] [longitude]}"] add_field = > [[geoip] [coordinates] " "% {[geoip] [latitude]}]} useragent {# parsing user browser characteristics source = >" User_Agent "target = >" ua "} mutate {# Log necessary processing convert = > [" [geoip] [coordinates] " "float"] convert = > ["Http_Status_Code", "integer"] convert = > ["Http_Bytes", "integer"] convert = > ["Request_Time", "float"] convert = > ["Upstream_Response_Time", "float"] remove_field = > ["beat", "@ version" "auth", "prospector", "source", "offset"] gsub = > ["timestamp", "\ + 0800", ""] # delete + 0800 of time gsub = > ["User_Agent", "\" ""] # remove the browser's double quotes} output {jdbc {connection_string = > "jdbc:mysql://mysql/elk?user=root&password=123456&useUnicode=true&characterEncoding=UTF8" driver_jar_path = > "/ usr/share/logstash/geodb/mysql-connector-java-8.0.18.jar" # idea to load mysql connector statement = > ["INSERT INTO nginx_access_log (Client_IP) Xerox ForwardedForceFortimestamps, appendices, versionreeds, DeviceDeviceDeviceGrads, DeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDevices, DeviceDeviceDeviceDeviceDeviceDevices, DeviceDevices, DeviceDevices, DeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceDeviceGraditions, VALUES, Client_IP, X_Forwarded_For, timestamp, etc.), "Client_IP", "X_Forwarded_For", "timestamp" "App_Version", "Device", "Device_Name", "System_Version", "Http_Method", "Site", "Http_Request", "Http_Status_Code", "Http_Referrer", "Usercode", "NetType", "Openudid", "Request_Time", "Upstream_Response_Time", "User_Agent", "[geoip] [city_name]", "[geoip] [region_name]"] # attention For example, you need to insert city information [geoip] [city_name]}}.
