Replace the original scribe service with flume

In the past, many businesses used scribe to support log collection, but then fb stopped supporting scribe. And the cost of compiling scribe on the machine is too high, and there are all kinds of holes, and it just so happens that flume has added support for scribe since 1.3.0. The data connected to the original scribe can be transferred to flume for collection. Although I like scribe very much, it is annoying to lose official support after all.

Agent.channels=c1agent.channels.c1.capacity=20000agent.channels.c1.transactionCapacity=10000agent.channels.c1.type=memoryagent.sinks=k1agent.sinks.k1.channel=c1agent.sinks.k1.hdfs.batchSize=8000agent.sinks.k1.hdfs.filePrefix=logagent.sinks.k1.hdfs.fileType=DataStreamagent.sinks.k1.hdfs.path=hdfs://NNHA/data/flume/% {category} /% Y%m%dagent.sinks.k1.hdfs.rollCount=0agent.sinks.k1.hdfs.rollInterval=86400agent.sinks.k1.hdfs.round=trueagent.sinks.k1.hdfs.roundUnit=minuteagent.sinks.k1.hdfs. RoundValue=1agent.sinks.k1.hdfs.serializer.appendNewline=falseagent.sinks.k1.hdfs.useLocalTimeStamp=trueagent.sinks.k1.hdfs.writeFormat=TEXTagent.sinks.k1.type=hdfsagent.sources=r1agent.sources.r1.channels=c1agent.sources.r1.host=0.0.0.0agent.sources.r1.port=1463agent.sources.r1.type=org.apache.flume.source.scribe.ScribeSourceagent.sources.r1.workerThreads=5

The main reason is that serializer.appendNewline is set to false, otherwise an enter will be automatically added to each entry, and there is not much else to explain. After using the natural second understanding of flume, in hdfs.path,% {category} means the category in the original scribe.

Flume 1.6's new features include support for kafka's source and sink, as well as regular filtering and delivery of data content, which is useful, and it looks as if a new book on flume will be released next month or the month after next.