Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

Install snappy for hadoop cdh version

Shulou Source: shulou.com Published: 2022-06-03 06:43:19 11月23日 Update

one。 Install protobuf

Ubuntu system

First, make sure that you have gcc gcc-c++ libtool cmake with these environments.

1 create a file libprotobuf.conf write content / usr/local/lib in the / etc/ld.so.conf.d/ directory, otherwise it will report error while loading shared libraries: libprotoc.so.8: cannot open shared obj

2../configure make&&make install

two。 Verify that installation is complete

Protoc-version

Libprotoc 2.5.0

two。 Install the snappy native library

Http://www.filewatcher.com/m/snappy-1.1.1.tar.gz.1777992-0.html

Download snappy-1.1.1.tar.gz

Decompress. / configure

Make&& makeInstall

Check / usr/local/lib

Libsnappy.a

Libsnappy.la

Libsnappy.so

Libsnappy.so.1

Libsnappy.so.1.2.0

three。 Compile the source code of cdh hadoop. (join snappy support)

Download link http://archive.cloudera.com/cdh6/cdh/5/

Hadoop-2.6.0-cdh6.11.0-src.tar.gz

Decompress. Compile with maven

4. Check the file

Hadoop-2.6.0-cdh6.11.0/hadoop-dist/target/hadoop-2.6.0-cdh6.11.0/lib/native

Whether there is a local library of hadoop and snappy under the directory

Copy the files under this directory to the lib/native directory under hadoop in the hadoop cluster and to the lib/native/Linux-amd64-64 directory under hbase. If not, create a new file, and each node needs to copy it.

Cp ~ apk/hadoop-2.6.0-cdh6.11.0/hadoop-dist/target/hadoop-2.6.0-cdh6.11.0/lib/native/* ~ / app/hadoop/lib/native/

6. Synchronize local libraries to other nodes

7. Configure core-site.xml for hadoop

Join

Io.compression.codecs

Org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec

Configure mapred-site.xml

Join

Mapreduce.map.output.compress

True

Mapreduce.map.output.compress.codec

Org.apache.hadoop.io.compress.SnappyCodec

Mapreduce.admin.user.env

LD_LIBRARY_PATH=/home/hadoop/app/hadoop/lib/native

Configure hbase-site.xml for hbase

Join

Hbase.block.data.cachecompressed

True

8. Restart hdfs and yarn of hadoop

9. Verify that the snappy is successful.

Hadoop checknative

18-03-07 17:33:36 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version

17:33:36 on 18-03-07 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library

Native library checking:

Hadoop: true / home/hadoop/app/hadoop/lib/native/libhadoop.so

Zlib: true / lib/x86_64-linux-gnu/libz.so.1

Snappy: true / home/hadoop/app/hadoop/lib/native/libsnappy.so.1

Lz4: true revision:10301

Bzip2: false

Openssl: true / usr/lib/x86_64-linux-gnu/libcrypto.so

See that snappy has successfully supported

Run the mapreduce task

Hadoop jar ~ / app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh6.11.0.jar wordcount / input/gisData / output

If it works properly. It proves that there is nothing wrong with snappy. If there is.

Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy () Z

Please check the local library configuration of mapred-site.xml

10 start hbase.

First create a table of snappy

Create 'snappyTest', {NAME= >' frank.Computation = > 'SNAPPY'}

Descibe 'snappyTest'

TTL = > 'FOREVER', COMPRESSION = >' SNAPPY', MIN_VERSIONS = >'0' just see the snappy.

The point is that we need to compress the existing tables.

Can be executed outside the shell

$echo "disable 'snappyTest2'" | hbase shell # disable the table

$echo "desc 'snappyTest2'" | hbase shell # View table structure

| $echo "alter 'snappyTest2', {NAME= >' fallow journal compression = > 'SNAPPY'}" | hbase shell # compression is modified to snappy |

$echo "enable 'snappyTest2'" | hbase shell # uses this table

$echo "major_compact 'snappyTest2'" | hbase shell # it is best to make the table region compact once

You can also hbase shell into shell manual compression. After compression, it will be found that the data has a compression ratio of about 40%.

The java code only needs to create the Hbase table

HColumnDescriptor HColumnDesc = new HColumnDescriptor ("data")

HColumnDesc.setCompressionType (Algorithm.SNAPPY); / / this sentence is the key

Tags: Directory configuration file check success copy node support compile run verify code task critical content manual data best source code environment Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno MySQL MariaDB Linux NVidia OPPO Reno