How to use gphdfs external table and hive together
This article mainly introduces the gphdfs external table and hive how to use, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let Xiaobian take you to understand.
Install java1.6 or above on all nodes
Install hadoop on segment and support the following versions.
Hadoop Distribution
Version
Gp_hadoop_ target_version
Pivotal HD
Pivotal HD 3.0, 3.0.1
Gphd-3.0
Pivotal HD 2.0, 2.1
Pivotal HD 1.0
Gphd-2.0
Greenplum HD
Greenplum HD 1.2
Gphd-1.2
Greenplum HD 1.1
Gphd-1.1 (default)
Cloudera
CDH 5.2, 5.3, 5.4.x, 5.5.x
Cdh6
CDH 5.0, 5.1
Cdh5.1
CDH 4.12-CDH 4.7
Cdh5.1
Hortonworks Data Platform
HDP 2.1, 2.2, 2.3
Hdp2
MapR
MapR 4.x
Gpmr-1.2
MapR 1.x, 2.x, 3.x
Gpmr-1.0
Apache Hadoop
2.x
Hadoop2
Configure the gp environment of all nodes
[gpadmin@db10 ~] $echo $JAVA_HOME
/ usr/java/default
[gpadmin@db10 ~] $echo $HADOOP_HOME
/ opt/hadoop
[gpadmin@db10 ~] $hadoop version
Hadoop 2.7.2
[gpadmin@db10 ~] $vim .bashrc
If [- f / etc/bashrc]; then
. / etc/bashrc
Fi
# User specific aliases and functions
Source / opt/greenplum-db/greenplum_path.sh
Export MASTER_DATA_DIRECTORY=/hdd1/master/gpseg-1
Export JAVA_HOME=/usr/java/default
Export HADOOP_HOME=/opt/hadoop
Export HADOOP_VERSION=2.7.2
Export PATH=$PATH:$JAVA_HOME/bin/:$HADOOP_HOME/bin
4. Configure gp parameters
[gpadmin@db9 ~] $gpconfig-c gp_hadoop_home-v "'/ opt/hadoop'"
[gpadmin@db9 ~] $gpconfig-c gp_hadoop_target_version-v "'hadoop2'"
[gpadmin@db9] $gpstop-u
5. Create tables on hive
Create table wdbd_dm.date_test1 (id int,name string)
Insert into wdbd_dm.date_test1 values (1 recording wjian')
Insert into wdbd_dm.date_test1 values (2 recorder wuj')
Insert into wdbd_dm.date_test1 values (3)
View the generated hdfs file on gp and generate 3 files
[gpadmin@db10 ~] $hdfs dfs-ls hdfs://172.20.5.8:8020/user/hive/warehouse/wdbd_dm.db/date_test1/
……
-rwxrwxr-x 3 root root 8 2017-05-25 16:02 hdfs://172.20.5.8:8020/user/hive/warehouse/wdbd_dm.db/date_test1/part-00000
-rwxrwxr-x 3 root root 6 2017-05-25 16:02 hdfs://172.20.5.8:8020/user/hive/warehouse/wdbd_dm.db/date_test1/part-00000_copy_1
-rwxrwxr-x 3 root root 7 2017-05-25 16:02 hdfs://172.20.5.8:8020/user/hive/warehouse/wdbd_dm.db/date_test1/part-00000_copy_2
Note: 172.20.5.8 is the namenode address of the target cluster, and 8020 is the port
Create the external table of gphdfs file
Create EXTERNAL table hdfs_test (id int,name varchar (32))
Location ('gphdfs://172.20.5.8:8020/user/hive/warehouse/wdbd_dm.db/date_test1/part*')
Format 'TEXT' (DELIMITER'\ 001')
Note: the default delimiter for hive files is the control character\ 001 of the ascii code.
Edw=# select * from hdfs_test
Id | name
-- +-
2 | wuj
3 | mike
1 | wjian
(3 rows)
Thank you for reading this article carefully. I hope the article "how to use gphdfs external form and hive" shared by the editor will be helpful to everyone. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!
