Shulou(Shulou.com)06/03 Report--

This article focuses on "performance comparison between Hadoop and spark". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn the performance comparison between Hadoop and spark.

What is the difference in performance between Hadoop and spark.

If Hadoop is a large contractor team, we can organize people to cooperate and move bricks to build houses, but the disadvantage is that it is slow.

Spark is another contractor team, which was set up late, but they are more flexible in moving bricks, can build houses interactively in real time, and are much faster than Hadoop.

When Hadoop starts to upgrade, assign dispatching expert YARN to dispatch workers. Spark moves bricks from multiple warehouses (HDFS,Cassandra,S3,HBase) and allows different experts such as YARN/ MESOS to schedule people and tasks.

Of course, when Spark works with the Hadoop team, the problem becomes more complicated. As two independent contractors, both have their own advantages and disadvantages and specific business use cases.

Therefore, we say that the performance difference between Hadoop and spark lies in:

Spark runs 100 times faster in memory than Hadoop and 10 times faster on disk. It is well known that Spark sorts 100TB data three times faster than Hadoop MapReduce on machines with only 1/10. In addition, Spark is also faster in machine learning applications, such as Naive Bayes and k-means.

The reason why Spark performs better than Hadoop is that every time you run a MapReduce task, Spark is not limited by input and output. As it turns out, applications are much faster. Then there is Spark's DAG, which can be optimized between steps. Hadoop does not have any periodic connections between MapReduce steps, which means that no performance tuning occurs at this level. However, if Spark runs on YARN with other shared services, performance may degrade and result in RAM overhead memory leaks. For this reason, Hadoop is considered to be a more efficient system if users have batch requests.

At this point, I believe you have a deeper understanding of the "performance comparison between Hadoop and spark". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.