Shulou(Shulou.com)06/01 Report--

This article is about how hive enables lzo compression. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Hive enable lzo compression 1.Hadoop install lzo dependency and configure

Involves the compilation of jar packages and the configuration of core-site files

You can set whether compression is enabled on the map or reduce side in the mapred-site file. Here you just set up the compression of mapreduce. Hadoop and hive can be set separately.

two。 Enable compression and set compression mode in hive

Set parameters in the hive client to start compression on the map or reduce side of hive.

Set mapred.output.compress=true;set hive.exec.compress.output=true;# must be LzopCodec to support sliced set mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec

3. Create a table to specify InputFormat and OutFormatCREATE EXTERNAL TABLE ods_log (`line`string) PARTITIONED BY (`dt`string) STORED AS INPUTFORMAT 'com.hadoop.mapred.DeprecatedLzoTextInputFormat' OUTPUTFORMAT' org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'LOCATION'/ warehouse/gmall/ods/ods_log'

INPUTFORMAT means that when reading from a table, reading from a lzo file is supported.

OUTPUTFORMAT refers to the format in which data is written to a file and input data

4. Upload data to table load data inpath'/ origin_data/gmall/log/topic_log/2020-06-14' into table ods_log partition (dt='2020-06-14')

After the data is imported into hive, the data is saved in the lzo file

5. Create an index for the lzo file hadoop jar / opt/module/hadoop-3.1.3/share/hadoop/common/hadoop-lzo-0.4.20.jar com.hadoop.compression.lzo.DistributedLzoIndexer / warehouse/gmall/ods/ods_log/dt=2020-06-14 Thank you for reading! This is the end of the article on "how to enable lzo compression in hive". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.