Hive installation record
Hive stand-alone mode installation-jared
The deployment note was recorded in early 2014 and is now on 51cto.
For more information on building the basic environment of hadoop, please refer to the following link:
Http://ganlanqing.blog.51cto.com/6967482/1387210
JDK version: jdk-7u51-linux-x64.rpm
Hadoop version: hadoop-0.20.2.tar.gz
Hive version: hive-0.12.0.tar.gz
Mysql driver package version: mysql-connector-java-5.1.7-bin.jar
1. Install the mysql environment
[root@master ~] # yum install mysql mysql-server-y
[root@master ~] # / etc/init.d/mysqld start
[root@master ~] # mysqladmin-uroot password "123456"
[root@master] # mysql-uroot-p123456
Welcome to the MySQL monitor. Commands end with; or\ g.
Your MySQL connection id is 2
Server version: 5.1.73 Source distribution
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
Affiliates. Other names may be trademarks of their respective
Owners.
Type 'help;' or'\ h' for help. Type'\ c'to clear the current input statement.
Mysql > create user 'hive' identified by' 123456'
Query OK, 0 rows affected (0.00 sec)
Mysql > GRANT ALL PRIVILEGES ON *. * TO 'hive'@'master' IDENTIFIED BY' 123456 'WITH GRANT OPTION
Query OK, 0 rows affected (0.00 sec)
Mysql > flush privileges
Query OK, 0 rows affected (0.00 sec)
Mysql > exit
Bye
[root@master ~] #
#
Missing steps for hive users to create hive libraries!
#
two。 Download the hive installation package
[jared@master conf] $wget http://mirror.bjtu.edu.cn/apache/hive/hive-0.12.0/hive-0.12.0-bin.tar.gz
[jared@master conf] $gzip-d hive-0.12.0.tar.gz
[jared@master conf] $tar-xf hive-0.12.0.tar
[jared@master conf] $mv hive-0.12.0 hive
3. Set environment variabl
[root@master ~] # vim / etc/profile
Export JAVA_HOME=/usr/java/jdk1.7.0_51
Export HIVE_HOME=/home/jared/hive
Export HIVE_CONF_DIR=/home/jared/hive/conf
Export HIVE_LIB=$HIVE_HOME/lib
Export HADOOP_INSTALL=/home/jared/hadoop
Export HBASE_INSTALL=/home/jared/hbase
Export PATH=$PATH:$HADOOP_INSTALL/bin:$HBASE_INSTALL/bin:$HIVE_HOME/bin
[root@master ~] # source / etc/profile
[root@master ~] # exit
Logout
[jared@master conf] $pwd
/ home/jared/hive/conf
[jared@master conf] $source / etc/profile
[jared@master conf] $echo $HIVE_HOME
/ home/jared/hive
[jared@master conf] $cp hive-env.sh.template hive-env.sh
[jared@master conf] $vim hive-env.sh
Export HADOOP_HEAPSIZE=1024
HADOOP_HOME=/home/jared/hadoop
Export HIVE_CONF_DIR=/home/jared/hive/conf
Export HIVE_AUX_JARS_PATH=/home/jared/hive/lib
[jared@master conf] $source hive-env.sh
4. Configure hive-site.xml
[jared@master conf] $vim hive-site.xml
Javax.jdo.option.ConnectionURL
Jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true
Javax.jdo.option.ConnectionDriverName
Com.mysql.jdbc.Driver
Javax.jdo.option.ConnectionUserName
Hive
Javax.jdo.option.ConnectionPassword
123456
5. Copy the driver package of mysql to the lib directory under the Hive installation path
[jared@master ~] $wget http://cdn.mysql.com/archives/mysql-connector-java-5.1/mysql-connector-java-5.1.7.tar.gz
[jared@master ~] $tar-zxvf mysql-connector-java-5.1.7.tar.gz
[jared@master ~] $cd mysql-connector-java-5.1.7
[jared@master ~] $cp mysql-connector-java-5.1.7-bin.jar / home/jared/hive/lib/
6.CLI access interface: shell
[jared@master ~] $hive
Logging initialized using configuration in jar bank filebank bank, home bank bank, hivebank bank, hivemercommonlue 0.12.0.jarbank bank, hiveMurray log4j.properties,
Hive > show databases
OK
Default
Time taken: 11.506 seconds, Fetched: 1 row (s)
Hive > create table test (key string)
OK
Time taken: 2.805 seconds
Hive > show tables
OK
Test
Time taken: 0.091 seconds, Fetched: 1 row (s)
Hive >
7. Local upload data test
Local file information
File name: access.log
Size: 11m
[jared@master input] $du-h access.log
11M access.log
[jared@master input] $cat access.log | wc-l
60000
Data structure:
[jared@master input] $cat access.log
1393960136.926 0 212.92.231.166 TCP_DENIED/403 1256 GET http://221.181.39.85/phpTest/zologize/axa.php-NONE/- text/html "-"--
1393960137.600 0 212.92.231.166 TCP_DENIED/403 1264 GET http://221.181.39.85/phpMyAdmin/scripts/setup.php-NONE/- text/html "-"--
1393960138.274 0 212.92.231.166 TCP_DENIED/403 1250 GET http://221.181.39.85/pma/scripts/setup.php-NONE/- text/html "-"--
1393960138.946 0 212.92.231.166 TCP_DENIED/403 1258 GET http://221.181.39.85/myadmin/scripts/setup.php-NONE/- text/html "-"--
1393960143.624 1 127.0.0.1 TCP_HIT/200 22874 GET http://www.chinacache.com/p_w_picpaths/logo.gif-NONE/- p_w_picpath/gif "-"--
1393960143.628 1 127.0.0.1 TCP_HIT/200 22874 GET http://www.chinacache.com/p_w_picpaths/logo.gif-NONE/- p_w_picpath/gif "-"--
1393960144.636 2 127.0.0.1 TCP_HIT/200 22874 GET http://www.chinacache.com/p_w_picpaths/logo.gif-NONE/- p_w_picpath/gif "-"--
1393960145.643 2 127.0.0.1 TCP_HIT/200 22874 GET http://www.chinacache.com/p_w_picpaths/logo.gif-NONE/- p_w_picpath/gif "-"--
1393982948.194 1 112.5.4.63 TCP_HIT/200 467 GET http://cu005.www.duba.net/duba/2011/kcomponent/kcom_commonfast/53a08fed.dat-NONE/- text/plain "-"--
1393982948.246 0 218.203.54.25 TCP_HIT/200 462GET http://cu005.www.duba.net/duba/2011/kcomponent/kcom_kvm2/indexkcom_kvm2.dat-NONE/- text/plain "-"--
1393982948.258 0 218.203.54.25 TCP_HIT/200 467 GET http://cu005.www.duba.net/duba/2011/kcomponent/kcom_commonfast/53a08fed.dat-NONE/- text/plain "-
Establish the table structure
Hive > CREATE TABLE CU005_LOG (TIMES_TAMP STRING,RES_TIME INT,FC_IP STRING,FC_HANDLING STRING,FILE_SIZE INT,REQ_METHOD STRING,URL STRING,USER STRING,BACK_SRC STRING,MIME STRING,REFERER STRING,UA STRING,COOKIE STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY''STORED AS TEXTFILE
Hive > show tables
OK
Cu005_log
Time taken: 0.08seconds, Fetched: 1 row (s)
Hive > desc cu005_log
OK
Times_tamp string None
Res_time int None
Fc_ip string None
Fc_handling string None
File_size int None
Req_method string None
Url string None
User string None
Back_src string None
Mime string None
Referer string None
Ua string None
Cookie string None
Time taken: 0.208 seconds, Fetched: 13 row (s)
Hive >
Import local data
Hive > LOAD DATA LOCAL INPATH'/ home/jared/input/access.log' OVERWRITE INTO TABLE CU005_LOG
Copying data from file:/home/jared/input/access.log
Copying file: file:/home/jared/input/access.log
Loading data to table default.cu005_log
Table default.cu005_log stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 10872324, raw_data_size: 0]
OK
Time taken: 1.811 seconds
Storage location in hadoop cluster
Hdfs://master:9000/user/hive/warehouse/cu005_log/access.log
[jared@master ~] $hadoop dfs-ls / user/hive/warehouse/
Found 1 items
Drwxr-xr-x-jared supergroup 0 2014-03-06 18:31 / user/hive/warehouse/cu005_log
Query
Hive > select count (*) from cu005_log
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
Set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
Set hive.exec.reducers.max=
In order to set a constant number of reducers:
Set mapred.reduce.tasks=
Starting Job = job_201402230829_0003, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201402230829_0003
Kill Command = / home/jared/hadoop/bin/../bin/hadoop job-kill job_201402230829_0003
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2014-03-06 17 1815 07994 Stage-1 map = 0%, reduce = 0%
2014-03-06 17 18 32121 Stage-1 map = 100%, reduce = 0%
2014-03-06 17 18 44 200 Stage-1 map = 100%, reduce = 33%
2014-03-06 17 18 47220 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201402230829_0003
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 HDFS Read: 10872324 HDFS Write: 6 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
60000
Time taken: 77.157 seconds, Fetched: 1 row (s)
Hive >
Web interface access
Please refer to https://cwiki.apache.org/confluence/display/Hive/HiveWebInterface for details.
You need to add some configuration items to the $HIVE_HOME/conf/hive_site.xml configuration file, as follows:
Hive.hwi.listen.host
192.168.255.25
This is the host address the Hive Web Interface will listen on
Hive.hwi.listen.port
9999
This is the port the Hive Web Interface will listen on
Hive.hwi.war.file
Lib/hive-hwi-0.12.0.war
This is the WAR file with the jsp content for Hive Web Interface
Start hive hwi
Background start
[jared@master] $nohup hive-- service hwi > / dev/null 2 > / dev/null &
Browsers access http://192.168.255.25:9999/hwi/
Use reference http://www.cnblogs.com/gpcuster/archive/2010/02/25/1673480.html
Comparison between HWI and CLI
If friends who have used cli read the above introduction, they will surely find a serious problem: there is no hint in the execution process. We don't know when a query execution ends.
Summarize the pros and cons of HWI versus CLI:
Advantages: HWI supports browser browsing, convenient and intuitive.
Disadvantages: no hint of execution process.
Personally, I prefer to use cli.
