Example Analysis of flume online configuration
This article mainly introduces the example analysis of flume online configuration, which has a certain reference value. Interested friends can refer to it. I hope you will learn a lot after reading this article.
Hdfs_agent.sources = R1
Hdfs_agent.sinks = K1
Hdfs_agent.channels = C1
# Describe/configure the source
Hdfs_agent.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
# hdfs_agent.sources.r1.zookeeperConnect = 10.69.40.17 2181 Magazine 10.69.40.12 Fringe 2181 Magazine 10.69.40.7 Fraser 2181
# hdfs_agent.sources.r1.groupId = online_kafka_to_hdfs_gk_groupid_000001
# hdfs_agent.sources.r1.topic = IRT_Dev_Status_STD
Hdfs_agent.sources.r1.kafka.bootstrap.servers = 10.69.40.10 9092Jing 10.69.40.13 Fringe 9092Jing 10.69.40.14 Fringe 9092Jing 10.69.40.15Rd 9092
Hdfs_agent.sources.r1.kafka.consumer.group.id = online_kafka_to_hdfs_gk_groupid_000001
Hdfs_agent.sources.r1.kafka.topics = workcondition_info_flume
Hdfs_agent.sources.r1.batchSize = 2000
Hdfs_agent.sources.r1.interceptors = i1
Hdfs_agent.sources.r1.interceptors.i1.type = host
# Describe the sink
Hdfs_agent.sinks.k1.type = hdfs
Hdfs_agent.sinks.k1.hdfs.path = hdfs://rootcloud/ods/%Y%m%d/%H
Hdfs_agent.sinks.k1.hdfs.filePrefix = log-% {host}
Hdfs_agent.sinks.k1.hdfs.fileType = DataStream
Hdfs_agent.sinks.k1.hdfs.rollInterval = 3600
Hdfs_agent.sinks.k1.hdfs.rollSize = 104857600
Hdfs_agent.sinks.k1.hdfs.rollCount = 0
Hdfs_agent.sinks.k1.hdfs.batchSize = 2000
Hdfs_agent.sinks.k1.hdfs.callTimeout = 300000
Hdfs_agent.sinks.k1.hdfs.idleTimeout = 600,
# Use a channel which buffers events in memory
Hdfs_agent.channels.c1.type = memory
Hdfs_agent.channels.c1.capacity = 2000000
Hdfs_agent.channels.c1.transactionCapacity = 3000
# Bind the source and sink to the channel
Hdfs_agent.sources.r1.channels = C1
Hdfs_agent.sinks.k1.channel = C1
Thank you for reading this article carefully. I hope the article "sample Analysis of flume online configuration" shared by the editor will be helpful to you. At the same time, I also hope that you will support and pay attention to the industry information channel. More related knowledge is waiting for you to learn!
