Kafka authentication configuration when submitting consumer kafka applications to spark cluster
Submitting a consuming kafka application contains sasl.jaas.config, which usually requires a configuration file. However, after the application is typed into a jar package, the configuration file in the jar package cannot be read through classload.
Add properties when you need to initialize kafka.
kafkaParams.put("sasl.jaas.config", "xxxx required\n"+ " accessKey=\"xxxx\"\n"+ " securityKey=\"xxxxx\";");
New lines, colons, quotation marks can not be less
Note that kafka depends on:
org.apache.kafka
kafka-clients
1.0.0
Also remove the kafka dependency from spark-streaming-kafka
org.apache.spark
spark-streaming-kafka-0-10_2.11
2.3.1
org.apache.kafka
kafka-clients