Shulou(Shulou.com)06/03 Report--

In each stage of MapReduce, different types of compression are enabled according to the scene, which can effectively improve the processing time of Job.

MR uses the various stages of compression:

Compression can be enabled at any stage of MapReduce, and the processing time of Job can be effectively reduced by compression.

From the above figure, it can be summarized as follows:

1. From the raw data to the Map phase; a splittable Splitable is required to start multiple threads to accelerate tasks

2.Map stage landing process; because Reduce needs to be carried out immediately, it is about Speed, so the sooner the better

The 3.Reduce phase is divided into results:

Archiving requirements: it is recommended to use high compression ratio tools to save disk space after archiving.

As input to the next task: to use Splitable, if not sliced, the next MapReduce will have only one task to handle the task

Comparison of the characteristics of compression tools:

Compression performance comparison:

Through the figure above, you can choose different compression tools according to different stages.

When choosing the compression tool, you can choose the corresponding compression tool from three aspects for different stages: compression ratio, compression speed, and whether to slice or not.

About the encoders introduced by compressed Hadoop:

Hadoop configuration of compression parameters:

Core-site.xml-related configuration: io.compression.codecs-> configure which compression formats are supported, and multiple compression formats are distinguished by commas

Mapred-site.xml-related configurations:

Reduce phase:

Mapreduce.output.fileoutputformat.compress: whether to enable compression. True means to enable it. Default is false.

Mapreduce.output.fileoutputformat.compress.codec: compression Typ

Map phase:

Mapreduce.map.output.compress: whether to enable compression. True means to enable it. Default is false.

Mapreduce.map.output.compress.codec: compression Typ

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.