Shulou(Shulou.com)06/01 Report--

Business process architecture diagram:

A set of data collection and analysis system based on logstash,redis,elasticSearch,kibana

Description of the architecture diagram:

Log collection system: (data source) the log recording behavior produced by the producer is collected and forwarded by logstash, then transmitted to redis for sequence, and finally displayed by ELK processing analysis.

ElasticSearch will store the data, index it (based on Lunce), establish a link to elasticSearch in kibana, and grab the indexed data in real time, so that the data can be displayed in real time. Through some data assembly and query conditions, we can get the results we want (can be displayed in various ways, such as charts, tables, line charts, etc.)

Effect picture:

#

1. ELK building

Http://dbspace.blog.51cto.com/6873717/1879106

II. Redis deployment

Note that redis must be above version 3.0, otherwise there will be all kinds of problems and holes will be encountered before.

Write the log collection configuration of logstash according to the architecture

[root@mycat conf] # cat / usr/local/logstash/conf/logstash_client.conf

Input {

File {

Type = > "log_count"

Path = > "/ var/log/messages"

}

}

Output {

Redis {

Host = > "192.168.1.114"

Port = > 6379

Data_type = > "list"

Key = > "key_count"

Db = > 0

}

}

Write the configuration of logstash to read data from redis and forward it to ELK

[root@mycat conf] # cat / usr/local/logstash/conf/logstash_server.conf

Input {

Redis {

Host = > "192.168.1.114"

Port = > 6379

Type = > "redis-input"

Data_type = > "list"

Key = > "key_count"

Db = > 0

}

}

Output {

Elasticsearch {

Hosts = > "192.168.1.114purl 9200"

Codec = > "json"

}

}

# the red parts should be the same

Start 2 services

/ usr/local/logstash/bin/logstash-f / usr/local/logstash/conf/logstash_server.conf &

/ usr/local/logstash/bin/logstash-f / usr/local/logstash/conf/logstash_client.conf &

5. Pie chart analysis

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.