In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-09-21 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >
Share
Shulou(Shulou.com)06/03 Report--
Spark can read data in text, csv and rmdb, and has automatic type detection
Public final static String DATA_SEPARATOR_TAB = "\ t"
Session.read () .format ("csv") .option ("delimiter", Constants.DATA_SEPARATOR_TAB) .option ("inferSchema", "true") .option ("header", "true") .option ("encoding", charset) .csv (path) .toDF (columnNames) .write () .mode (mode) .saveAsTable (tempTable)
The red attribute determines whether spark automatically detects the data type. If automatic detection is not enabled, the default is string.
By default, type detection and correspondence will occur when rdbms is imported into spark, but there is a problem with the time type of processing sqlserver.
As shown in the above figure
Only datetime can be recognized and stored as a date type by spark, and others fall into String type, so the performance will be much worse when performing data_formate because of one more step of conversion.
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
The market share of Chrome browser on the desktop has exceeded 70%, and users are complaining about
The world's first 2nm mobile chip: Samsung Exynos 2600 is ready for mass production.According to a r
A US federal judge has ruled that Google can keep its Chrome browser, but it will be prohibited from
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
About us Contact us Product review car news thenatureplanet
More Form oMedia: AutoTimes. Bestcoffee. SL News. Jarebook. Coffee Hunters. Sundaily. Modezone. NNB. Coffee. Game News. FrontStreet. GGAMEN
© 2024 shulou.com SLNews company. All rights reserved.