In non-kerberos environment, flume collects logs to kafka

Kafkaflume.conf

Agent.sources = s1agent.channels = c1agent.sinks = k1agent.sources.s1.type=execagent.sources.s1.command=tail-F / usr/local/src/flume/testflume2.logagent.sources.s1.channels=c1agent.channels.c1.type=memoryagent.channels.c1.capacity=10000agent.channels.c1.transactionCapacity=100# set Kafka receiver # agent.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink## set broker address and port number of Kafka agent.sinks.k1.kafka.bootstrap.servers = master:9092,node01:9092 Node02:9092## sets Topicagent.sinks.k1.kafka.topic of Kafka = mydemo5## sets serialization mode # # agent.sinks.k1.kafka.flumeBatchSize = 20agent.sinks.k1.kafka.producer.acks = 1agent.sinks.k1.kafka.producer.linger.ms = 1agent.sinks.ki.kafka.producer.compression.type = snappyagent.sinks.k1.channel=c1

Create a topic

Kafka-topics-create-zookeeper master:2181/kafka-replication-factor 2-partitions 3-topic mydemo5

Start flume as a message producer to write on kafka

Flume-ng agent-c conf-f kafkaflume.conf-n agent-Dflume.root.logger=INFO,console

Consume kafka data

Kafka-console-consumer-bootstrap-server master:9092-topic mydemo5-from-beginning

Read the contents of a file and write to append to a new file

#! / bin/bashwhile read linedo echo $line sleep 0.01echo-e $line > > / usr/local/src/flume/testflume2.logdone < / etc/sudo-ldap.conf