Shulou(Shulou.com)05/31 Report--

This article shares with you what to do if the spark cluster cannot allocate resources. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Question 1:WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

At this point, the program will always loading,running,loading,running. And continue to appear the above warning.

This error is most likely due to the way you started spark.

Many people use start-all.sh directly to launch spark, but if you do not specify the ip of master in the env configuration file, startup will report the above problem. If there is no configuration, you can specify it at startup:

Use on maser

. / start-master.sh-h "ip address"

Use on worker

. / start-slave.sh spark:// "ip of master": 7077

Other parameters can also be specified at startup, and the port number can also be changed.

Thank you for reading! This is the end of the article on "what to do if the spark cluster cannot allocate resources". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.