[Druid] simple configuration process for Druid to read Kafka data
Druid standalone version installation reference: blog.51cto.com/10120275/2429912
Druid real-time access to Kafka process
Download, install and launch kafka process:
wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.2.1/kafka_2.11-2.2.1.tgztar -zxvf kafka_2.11-2.2.1.tgzln -s kafka_2.11-2.2.1 kafka$KAFKA_HOME/kafka-server-start.sh ~/kafka/config/server.properties 1>/dev/null 2>&1 &
Creating a topic: Wikipedia
./ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic wikipedia
Unzip wikiticker-2015-09-12-sampled.json.gz file, this step is to prepare the input file for kafka topic
cd $DRUID_HOME/quickstart/tutorialgunzip -k wikiticker-2015-09-12-sampled.json.gz
After this step is completed, generate wikiticker-2015-09-12-sampled.json under the $DRIUID_HOME/quickstart/tutorial folder
上图配置文件如下，其中bootstrap.servers配置kafka地址
{ "type": "kafka", "dataSchema": { "dataSource": "wikipedia", "parser": { "type": "string", "parseSpec": { "format": "json", "timestampSpec": { "column": "time", "format": "auto" }, "dimensionsSpec": { "dimensions": [ "channel", "cityName", "comment", "countryIsoCode", "countryName", "isAnonymous", "isMinor", "isNew", "isRobot", "isUnpatrolled", "metroCode", "namespace", "page", "regionIsoCode", "regionName", "user", { "name": "added", "type": "long" }, { "name": "deleted", "type": "long" }, { "name": "delta", "type": "long" } ] } } }, "metricsSpec" : [], "granularitySpec": { "type": "uniform", "segmentGranularity": "DAY", "queryGranularity": "NONE", "rollup": false } }, "tuningConfig": { "type": "kafka", "reportParseExceptions": false }, "ioConfig": { "topic": "wikipedia", "replicas": 2, "taskDuration": "PT10M", "completionTimeout": "PT20M", "consumerProperties": { "bootstrap.servers": "localhost:9092" } }}
接下来要将wikiticker-2015-09-12-sampled.json文件内容，利用kafka生产者脚本写入wikipedia的topic中
export KAFKA_OPTS="-Dfile.encoding=UTF-8"./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wikipedia < {PATH_TO_DRUID}/quickstart/tutorial/wikiticker-2015-09-12-sampled.json
