Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

Kafka authentication configuration when submitting consumer kafka applications to spark cluster

Shulou Source: shulou.com Published: 2022-06-03 07:25:14 11月23日 Update

Submitting a consuming kafka application contains sasl.jaas.config, which usually requires a configuration file. However, after the application is typed into a jar package, the configuration file in the jar package cannot be read through classload.

Add properties when you need to initialize kafka.

kafkaParams.put("sasl.jaas.config", "xxxx required\n"+ " accessKey=\"xxxx\"\n"+ " securityKey=\"xxxxx\";");

New lines, colons, quotation marks can not be less

Note that kafka depends on:

org.apache.kafka

kafka-clients

1.0.0

Also remove the kafka dependency from spark-streaming-kafka

org.apache.spark

spark-streaming-kafka-0-10_2.11

2.3.1

org.apache.kafka

kafka-clients

Tags: Applications configuration files consumption colons at the same time attributes quotes questions clusters Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno Shulou Technology MySQL NVidia MariaDB OPPO Reno