Shulou(Shulou.com)06/03 Report--

Recently, after listening to teacher Wang Jialin's 2016 big data Spark "Mushroom Cloud" campaign, we need to integrate flume,kafka and Spark streaming. It's hard to get started at the moment, so let's start with simplicity:

1. Netcat mode + Logger log display

The Flume configuration file is as follows:

# example.conf: a single-node Flume configuration# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = centering Describe/configure the sourcea1.sources.r1.type = netcata1.sources.r1.bind = localhosta1.sources.r1.port = 44444th Describe the sinka1.sinks.k1.type = logger# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 10mm Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = C1

Startup of flume:

$bin/flume-ng agent-- conf conf--conf-file example.conf-- name A1-Dflume.root.logger=INFO,console

Trigger data:

Telnet localhost 44444

The same character appears on the console of flume.

Second, flume cascade mode

Through avro, data is transferred between two machines, and the source is netcat

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.