Fluentd parses httpd log
Need to parse the log of httpd
107.106.8.237.145-[12/Feb/2020:15:09:52 + 0800] "GET / noindex/css/fonts/Light/OpenSans-Light.ttf HTTP/1.1" 404 240 "http://39.96.45.213/noindex/css/open-sans.css"" Mozilla/5.0 (Macintosh; Intel Mac OS X 10 / 12 / 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36 "
Fluentd configuration
@ id httpd @ type tail path / data/logs/kim-todo-api-prod/log/statistics.log pos_file / home/web_server/ingress-nginx/statistics.log.pos tag * read_from_head true format / ^ (? [^] *) [^] * (? [^] *)\ [(? [^\]] *)\] "(?\ S+) (: + (? [^] *) +\ S*)? (? [ ^] *) (? [^] *) (?: "(? [^\"] *) "(? [^\"] *) ")? $/
The effect after analysis
1. Format is followed by a regular matching statement
2. () means to capture a packet. () will save the matching values in each packet. It is not difficult to see that there is a key in angle brackets, and the regular match in parentheses () matches the desired value.
3. (?:) represents a non-capture group. the only difference between a capture packet and a capture packet is that the matching value of the non-capture packet will not be saved. Here you will wonder that the characters in (?:) are still matched. There are several special uses here.
Non-capture arrays do not participate in number allocation but participate in matching
A: in the form of (?: (exp))
That is to say, grouping according to normal grouping logic in exp
B: shaped like (?: () | ())
Strictly speaking, writing like this is a way to solve the problem of choosing caching. (using parentheses directly has a side effect, which is that the relevant matches are cached, which is available at this time?: put before the first option to eliminate this side effect. )
C: like ((?: exp))
I think this writing is equivalent to (exp)
4. Parentheses [] outside parentheses remove some useless characters
5. The right slash\ is an escape character.
6.\ s means to match if it is not blank + means to match the previous subexpression one or more times
7. [^] * means to match all empty characters
8 、? Indicates a non-greedy match, that is, if a match reaches a match, ok will not continue to match.
9. The expression begins and ends with "indicates that what matches" is the content in "."
10, + means to match the previous subexpression one or more times
