Shulou(Shulou.com)06/03 Report--

Spark is a distributed memory computing framework, rather than a distributed content storage framework, with tachyon distributed memory file system, it will be more effective.

Spark is 10 times faster than hadoop in file mode and 100 times faster in memory computing mode.

Here are some

1 spark is a distributed memory-based computing framework that is especially suitable for iterative computing.

2 mapReduce has two stages: map and reduce, while spark is constantly iterative, which makes it more flexible and powerful, and it is easy to construct complex algorithms.

3 spark can not replace hive,hive for data warehouse storage, spark sql only replaces hive's computing engine as the analysis platform, and hadoop as the storage platform.

4 spark intermediate data can be in memory or on disk

5 partition is a data collection

6 Note: beginners should pay attention to step-by-step inspection when performing multiple steps, otherwise they don't know where to go wrong.

7 var data = sc.textFile ("/ user") do not need to write hdfs://, judge according to the context

8 read the file to get HadoopRDD, remove the index in the file, get MapPartitionsRDD, so a series of fragmented data are distributed on different machines.

9 Mobile computing instead of mobile data

Mobile computing means that it is distributed to each machine and calculates the part of the data it is responsible for, rather than moving the data from each machine to calculate.

After the calculation, the data are summarized.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.