Shulou(Shulou.com)06/03 Report--

==> Cache data in memory

---> Performance tuning is mainly to put data into memory operation

---> Examples of use:

//Read data from Oracle database to generate DataFrameval oracleDF = spark.read.format("jdbc") .option("url", "jdbc:oracle:thin:@192.168.10.100:1521/orcl.example.com") .option("dbtable", "scott.emp") .option("user", "scott") .option("password", "tiger").load //Register DataFrame as table oracleDF.registerTempTable("emp")//Execute query and monitor execution time through Web Console spark.sql ("select * from emp").show//Cache table and query twice and monitor execution time through Web Console spark.sqlContext.cacheTable("emp")//Empty cache spark.sqlContext. cacheTable("emp")spark.sqlContext.clearCache

==> Optimize relevant parameters

---> spark.sql.inMemoryColumnarStorage.compressed

---Default value: true

Spark SQL will automatically select a compression encoding for each column based on statistics

---> spark.sql.inMemoryColumnarStorage.batchSize

---Default value: 10000

---Cache batch size, larger batches can improve memory utilization and compression, but also bring OOM(Out Of Memory) risk

---> spark.sql.files.maxPartitionBytes

---Default value: 128M

----Maximum number of bytes a single partition can hold when reading a file

---> spark.sql.files.openCostinBytes

---Default value: 4M

---Estimated cost of opening files, measured in bytes that can be scanned at the same time, used when writing multiple files to a partition, overestimation is relatively good, so small file partitions will be faster than large file partitions (priority scheduling)

---> spark.sql.autoBroadcastJoinThreshold

---Default value: 10M

----Used to configure the maximum byte size that a table can broadcast to all worker nodes when performing a join operation. Generally, setting this value to-1 disables broadcasting.

---Note: The current statistics only support Hive Metastore tables that have run the ANALYZE TABLE COMPUTE STATISTICS noscan command.

---> spark.sql.shuffle.partitions

---Default value: 200

----Number of partitions used to configure join or aggregate operations to shuffle data

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.