Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

In non-kerberos environment, flume collects logs to kafka

Shulou Source: shulou.com Published: 2022-06-02 19:07:59 11月18日 Update

Kafkaflume.conf

Agent.sources = s1agent.channels = c1agent.sinks = k1agent.sources.s1.type=execagent.sources.s1.command=tail-F / usr/local/src/flume/testflume2.logagent.sources.s1.channels=c1agent.channels.c1.type=memoryagent.channels.c1.capacity=10000agent.channels.c1.transactionCapacity=100# set Kafka receiver # agent.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink## set broker address and port number of Kafka agent.sinks.k1.kafka.bootstrap.servers = master:9092,node01:9092 Node02:9092## sets Topicagent.sinks.k1.kafka.topic of Kafka = mydemo5## sets serialization mode # # agent.sinks.k1.kafka.flumeBatchSize = 20agent.sinks.k1.kafka.producer.acks = 1agent.sinks.k1.kafka.producer.linger.ms = 1agent.sinks.ki.kafka.producer.compression.type = snappyagent.sinks.k1.channel=c1

Create a topic

Kafka-topics-create-zookeeper master:2181/kafka-replication-factor 2-partitions 3-topic mydemo5

Start flume as a message producer to write on kafka

Flume-ng agent-c conf-f kafkaflume.conf-n agent-Dflume.root.logger=INFO,console

Consume kafka data

Kafka-console-consumer-bootstrap-server master:9092-topic mydemo5-from-beginning

Read the contents of a file and write to append to a new file

#! / bin/bashwhile read linedo echo $line sleep 0.01echo-e $line > > / usr/local/src/flume/testflume2.logdone < / etc/sudo-ldap.conf

Tags: File content slogan address sequence receiver data method message producer consumption production log environment Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno vpn MariaDB NVidia Microsoft macOS