Flume case support

Friday, 2019-2-22

Flume case support

The deployment type of flume / / reference link is https://www.cnblogs.com/qingyunzong/p/8994494.html

1. Single process

2. Multi-agent process (multiple agent sequential links) it is not recommended that if there is a failure, the whole event event fails.

3. Merging of streams (data from multiple Agent converge to the same Agent) / / A wide range of application scenarios

4. Multilevel streams (multilevel streams) / / Flume also supports multilevel streams, what multilevel streams? For example, when log streams mixed with syslog, java, nginx, tomcat, and so on start to flow into an agent, you can separate the mixed log streams in agent and establish its own transmission channel for each log.

5. Load balance function / / Agent1 is a routing node, which is responsible for balancing the Event of Channel temporary storage to the corresponding multiple Sink components, and each Sink component is connected to an independent Agent.

Flume action

Flume working mechanism

Flume architecture, components

Common configuration of flume

Flume Sources description

Avro source: Avro is a data serialization system designed for applications that support mass data exchange

Avro can send a given file to the Flume,Avro source using the AVRO RPC mechanism.

ThriftSource: ThriftSource is basically the same as Avro Source. Just change the type of source to thrift, for example, a1.sources.r1.type = thrift, which is relatively simple.

ExecSource: the configuration of ExecSource is to set a Unix (linux) command, and then use this command to continuously output data. If the process exits, Exec Source exits with it, and no further data is generated. / / EXEC executes a given command to get the source of the output. If you want to use the tail command, you must make the file large enough to see the output.

JMS Source: reading data from the JMS system (messages, topics)

Spooling Directory Source:Spooling Directory Source monitors the new files in the configured directory and reads the data in the files.

Among them, Spool Source has two points for attention, the first is that the files copied to the spool directory can no longer be opened and edited, and the second is that the spool directory cannot contain the corresponding subdirectories. This is mainly used for quasi-real-time monitoring of logs.

The port on which Syslogtcp Source:Syslogtcp listens on TCP is used as the data source

JSONHandler Source:

Sink

Hadoop sink: eventually store the data on the hdfs

File Roll Sink: stores events in the local file system. Generate a file at specified intervals to save log information collected during this period of time.

Channel

Replicating Channel Selector:

Flume supports Fan out streams from one source to multiple channels. There are two modes of Fan out, replication and reuse.

In the case of replication, the events of the stream are sent to all configured channels.

In the case of reuse, events are sent to a subset of the available channels.

The Fan out flow needs to specify the rules for the source and Fan out channel.

Memory Channel event is saved in Java Heap. If a small amount of data is allowed to be lost, it is recommended to use

File Channel event is saved in a local file with high reliability but lower throughput than Memory Channel.

JDBC Channel event is stored in relational data and is generally not recommended.

Getting started with Flume-several different Sinks https://www.cnblogs.com/itdyb/p/6270893.html

Source type https://www.cnblogs.com/qingyunzong/p/8995554.html of Flume

Flume: three kinds of channel compare https://blog.csdn.net/weixin_38750084/article/details/83059871