In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-09-24 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >
Share
Shulou(Shulou.com)06/01 Report--
This article is about the use of Spark2.3.1 skills to share with you, the editor feels very practical, so share with you to learn, I hope you can learn something after reading this article, do not say much, follow the editor to have a look.
Spark 2.3.1 use the technique Spark-SQL to read the JSON file when reflecting the header case class StudentInfo (id:Long,name:String,age:Int) val example = spark.read.json ("/ data/result.json") .as (StudentInfo) example.show () dynamically define schema
Different schema needs to be defined according to different data
Val schemaInfo = "name age" val fields = schemaInfo.map (item= > item.split (") .map (item= > StructField (item,StringType,nullable=true)) val schema = StructType (fields) val rowRDD = peopleRDD.map (_ .split (") .map (attributes= > Row (attributes (0), attributes (1)) val peopleDF = spark.createDataFrame (rowRDD,schema) peopleDF.show () Spark 2.3.1 on YARNspark-submit limit parameter is not valid
Because the executor-memory 2g configured during spark-submit did not take effect, he later asked his colleague that he also encountered the same problem, and the solution was to allocate executor dynamically.
-- conf spark.yarn.maxAppAttempts=1-conf spark.yarn.maxAppAttempts=1-- conf spark.dynamicAllocation.minExecutors=2-- conf spark.dynamicAllocation.maxExecutors=4-- the above is what the skills of using Spark2.3.1 are, and the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
The market share of Chrome browser on the desktop has exceeded 70%, and users are complaining about
The world's first 2nm mobile chip: Samsung Exynos 2600 is ready for mass production.According to a r
A US federal judge has ruled that Google can keep its Chrome browser, but it will be prohibited from
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
About us Contact us Product review car news thenatureplanet
More Form oMedia: AutoTimes. Bestcoffee. SL News. Jarebook. Coffee Hunters. Sundaily. Modezone. NNB. Coffee. Game News. FrontStreet. GGAMEN
© 2024 shulou.com SLNews company. All rights reserved.