How to use Sink of Flume
This article mainly introduces "how to use Sink of Flume". In daily operation, I believe many people have doubts about how to use Sink of Flume. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how to use Sink of Flume". Next, please follow the editor to study!
Logger Sink
Logs is output to console, which is used for debug.
[root@hftest0001 conf] # pwd/opt/apache-flume-1.6.0-bin/conf [root@hftest0001 conf] # vi s-exec_c-m_s-logger.conf agent.sources = exec_tailagent.channels = memoryChannelagent.sinks = loggerSinkagent.sources.exec_tail.type = execagent.sources.exec_tail.command = tail-F / opt/flume-data/exec-tail.logagent.sources.exec_tail.channels = memoryChannelagent.sinks.loggerSink.type = loggeragent.sinks.loggerSink.channel = memoryChannelagent Type = memoryagent.channels.memoryChannel.capacity = 100 [root@hftest0001 apache-flume-1.6.0-bin] # pwd/opt/apache-flume-1.6.0-bin [root@hftest0001 opt] # mkdir-p / opt/flume-data/ [root@hftest0001 opt] # touch / opt/flume-data/exec-tail.log [root@hftest0001 apache-flume-1.6.0-bin] # flume-ng agent-n agent-c conf/-f conf/s- Exec_c-m_s-logger.conf. [root@hftest0001 opt] # echo "Hello Flume" > > / opt/flume-data/exec-tail.log observe console Similar to Event: {headers {} bodys: xxx Hello Flume}
HDFS Sink
[root@hftest0001 conf] # pwd/opt/apache-flume-1.6.0-bin/conf [root@hftest0001 conf] # vi s-exec_c-m_s-hdfs.conf agent.sources = exec_tail agent.channels = memoryChannelagent.sinks = hdfs_sinkagent.sources.exec_tail.type = execagent.sources.exec_tail.command = tail-F / opt/flume-data/exec-tail.logagent.sources.exec_tail.interceptors = i1agent.sources.exec_tail.interceptors.i1.type = Org.apache.flume.interceptor.TimestampInterceptor$Builderagent.sources.exec_tail.channels = memoryChannelagent.sinks.hdfs_sink.type = hdfsagent.sinks.hdfs_sink.hdfs.path = hdfs://10.224.243.124:9000/flume/events/%y-%m-%d = > three policies for writing the path to hdfs # # roll file (avoid generating a large number of empty files, or small files) # agent.sinks.hdfs_sink.hdfs.rollInterval = 30 = > based on time: default 30s, set to 0, disable#agent.sinks.hdfs_sink.hdfs.rollSize = 1024 = > based on file size: default 1024bytes, set to 0, disable#agent.sinks.hdfs_sink.hdfs.rollCount = 10 = > number of file messages: default 10, set to 0 Then disableagent.sinks.hdfs_sink.hdfs.fileType = DataStream = > flume file type written to hdfs default:SequenceFile#SequenceFile = > similar to hadoop.io.LongWritable "ora.apache.hadoop.io.ByteWritable...". # DataStream = > will not compress the output, that is, if you can no longer set hdfs.codeC#CompressedStream = >, the output will be compressed And it is required to set the available hdfs.codeC#agent.sinks.hdfs_sink.hdfs.codeCagent.sinks.hdfs_sink.hdfs.writeFormat = Textagent.sinks.hdfs_sink.hdfs.filePrefix = flume#agent.sinks.hdfs_sink.hdfs.hdfs.callTimeout = 10000 = > the operation of flume on Hdfs, such as open,write,flush, etc. For systems with poor network, the parameter agent.sinks.hdfs_sink.channel = memoryChannelagent.channels.memoryChannel.type = memoryagent.channels.memoryChannel.capacity = 100 can be appropriately increased.
File Roll Sink
[root@hftest0001 conf] # pwd/opt/apache-flume-1.6.0-bin/conf [root@hftest0001 conf] # vi s-exec_c-m_s-file-roll.conf agent.sources = exec_tail-1agent.channels = memoryChannelagent.sinks = file_roll-1agent.sources.exec_tail-1.type = execagent.sources.exec_tail-1.command = tail- F / opt/flume-data/exec-tail.logagent.sources.exec_tail-1.channels = memoryChannelagent.sinks .file _ roll-1.type = file_rollagent.sinks.file_roll-1.sink.directory= / opt/flume-data/file-roll-1#agent.sinks.file_roll-1.sink.rollInterval= 30 = > roll file policy Default:30s generates a new file. Set to 0, then disable roll file, that is, all will be written to a single file agent.sinks.file_roll-1.channel = memoryChannelagent.channels.memoryChannel.type = memoryagent.channels.memoryChannel.capacity = 100. at this point, the study on "how to use Flume's Sink" is over. I hope to be able to solve everyone's doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!
