How to enable Compression of Intermediate data in Hive

This article mainly introduces how to enable compression of intermediate data in Hive, which has a certain reference value, which can be used for reference by interested friends. I hope you will gain a lot after reading this article.

Enable compression for intermediate data

Complex Hive queries are usually transformed into a series of multi-stage MapReduce jobs, and these jobs are linked by the Hive engine to complete the entire query. Therefore, the "intermediate output" here refers to the output of the previous MapReduce job, which will be used as input data for the next MapReduce job.

Compression can significantly reduce the amount of intermediate data, thus internally reducing the amount of data transfer between Map and Reduce.

We can use the following properties to enable compression on the intermediate output.

Set hive.exec.compress.intermediate=true

Set hive.intermediate.compression.codec=org.apache.hadoop.io.compress.SnappyCodec

Set hive.intermediate.compression.type=BLOCK

To compress the data that is finally output to HDFS, you can use the following properties:

Set hive.exec.compress.output=true

Here are some compression codecs that can be used

Org.apache.hadoop.io.compress.DefaultCodec

Org.apache.hadoop.io.compress.GzipCodec

Org.apache.hadoop.io.compress.BZip2Codec

Com.hadoop.compression.lzo.LzopCodec

Org.apache.hadoop.io.compress.Lz4Codec

Org.apache.hadoop.io.compress.SnappyCodec thank you for reading this article carefully. I hope the article "how to enable compression of intermediate data in Hive" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!