Introduction to the core concepts of flume
The main content of this article is "introduction to the core concepts of flume". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "introduction to the core concepts of flume".
Core concept
Event
Client
Agent
Sources 、 Channels 、 Sinks
Other components: Interceptors, Channel Selectors, Sink Processor
Core concept: Event
Event is the basic unit of Flume data transmission. Flume transmits data from the source to the final destination in the form of events. The Event consists of an optional hearders and a byte array containing data.
The data contained is opaque to flume
Headers is an unordered collection containing pairs of key-value strings, and key is unique within the collection.
Headers can use extensions in context routing
Public interface Event {
Public Map getHeaders ()
Public void setHeaders (Map headers)
Public byte [] getBody ()
Public void setBody (byte [] body)
}
Core concept: Client
Clinet is an entity that wraps the original log into events and sends them to one or more agent.
For example
Flume log4j Appender
You can customize a specific Client using Client SDK (org.apache.flume.api)
The aim is to decouple Flume from the data source system
Is not required in the topology of flume
Core concept: Agent
An Agent contains Sources, Channels, Sinks, and other components that it uses to transfer events from one node to another or the final destination.
Agent is the basic part of the flume stream.
Flume provides configuration, life cycle management, and monitoring support for these components.
Core concept: Source
Source is responsible for receiving events or generating events through a special mechanism, and placing events in batches on one or more Channels. There are two types of Source: event driver and polling.
Different types of Source:
And the well-known system integration Sources: Syslog, Netcat
Sources: Exec, SEQ for automatically generated events
IPC Sources: Avro for communication between Agent and Agent
Source must be associated with at least one channel
Core concept: Channel
Channel is located between Source and Sink and is used for cached events. When Sink successfully sends events to the next hop's channel or final destination, events is removed from Channel.
Different Channels provides different levels of persistence:
Memory Channel: volatile
File Channel: based on WAL (pre-written log Write-Ahead Logging) implementation
JDBC Channel: implementation based on embedded Database
Channels supports transactions
Provide a weaker sequence guarantee
Can work with any number of Source and Sink
Core concept: Sink
Sink is responsible for transferring the events to the next hop or final destination, and removes the events from the channel upon successful completion.
Different types of Sinks:
Store events to the terminal Sink of the final destination. For example: HDFS, HBase
Automatic consumption of Sinks. For example: Null Sink
IPC sink: Avro for inter-Agent communication
Must work with an exact channel
Core concept: Interceptor
A set of Interceptor for Source that decorates and filters the events where necessary in a preset order.
The built-in Interceptors allows you to add event headers such as timestamp, hostname, static tag, etc.
A custom interceptors can create a specific headers where necessary by introspecting the event payload (reading the original log).
Core concept: Channel Selector
Channel Selector allows Source to select one or more Channel from all Channel based on preset criteria
Built-in Channel Selectors:
Copy Replicating: event is copied to the relevant channel
Multiplexing Multiplexing: routed to a specific channel based on hearder,event
Core concept: Sink Processor
Multiple Sink can form a Sink Group. A Sink Processor is responsible for activating a Sink from a specified Sink Group. Sink Processor can achieve load balancing through all the Sink in the group, or it can be transferred to another Sink if it fails.
Flume implements load balancing (Load Balancing) and failover (failover) through Sink Processor
Built-in Sink Processors:
Load Balancing Sink Processor-use RANDOM, ROUND_ROBIN, or custom selection algorithms
Failover Sink Processor
Default Sink Processor (single Sink)
All Sink are polled (polling) to get events from Channel. This action is activated by Sink Runner
Sink Processor acts as an agent for Sink
At this point, I believe you have a deeper understanding of the "introduction to the core concepts of flume". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!
