Hadoop--4.Hadoop installation configuration
First, download
1. Download hadoop
[root@master ~] # cd / usr/local/src/
[root@master src] # wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
two。 Download address on the official website (latest available)
Http://hadoop.apache.org/releases.html#Download
two。 Installation
1. Decompression
[root@master src] # tar zxvf hadoop-2.7.2.tar.gz
two。 move
[root@master src] # mv hadoop-2.7.2 / usr/local/
3. Copy to all slave
[root@master src] # rsync-av / usr/local/hadoop slave1:/usr/local/
[root@master src] # rsync-av / usr/local/hadoop slave2:/usr/local/
[root@master src] # rsync-av / usr/local/hadoop slave3:/usr/local/
[root@master src] # rsync-av / usr/local/hadoop slave4:/usr/local/
[root@master src] # rsync-av / usr/local/hadoop slave5:/usr/local/
[root@master src] # rsync-av / usr/local/hadoop slave6:/usr/local/
III. Configuration
1. Create a directory
[root@master] # mkdir-p / data/hadoop
[root@master ~] # cd! $& & mkdir tmp dfs dfs/data dfs/name
Do the same on all slave.
two。 Configure the core-site.xml file
[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/core-site.xml
Add the following:
Fs.defaultFS
Hdfs://192.168.0.194:9000
Hadoop.tmp.dir
File:/data/hadoop/tmp/
Io.file.buffer.size
131702
Note:
The IP after ① hdfs is the ip of master.
The path followed by ② file is to create the tmp directory path
3. Configure the hdfs-site.xml file
[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/hdfs-site.xml
Add the following:
Dfs.namenode.name.dir
File:/data/hadoop/dfs/name
Dfs.datanode.data.dir
File:/data/hadoop/dfs/data
Dfs.replication
two
Dfs.namenode.secondary.http-address
192.168.0.194:9001
Dfs.webhdfs.enabled
True
4. Configure the mapred-site.xml file
[root@master hadoop] # cp / usr/local/hadoop/etc/hadoop/mapred-site.xmll.template / usr/local/hadoop/etc/hadoop/mapred-site.xml
[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/mapred-site.xml
Add the following:
Mapreduce.framework.name
Yarn
Mapreduce.jobhistory.address
192.168.0.194:10020
Mapreduce.jobhistory.webapp.address
192.168.0.194:19888
5. Configure the yarn-site.xml file
[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/yarn-site.xml
Add the following:
Yarn.nodemanager.aux-services
Mapreduce_shuffle
Yarn.nodemanager.auxservices.mapreduce.shuffle.class
Org.apache.hadoop.mapred.ShuffleHandler
Yarn.resourcemanager.address
192.168.0.194:8032
Yarn.resourcemanager.scheduler.address
192.168.0.194:8030
Yarn.resourcemanager.resource-tracker.address
192.168.0.194:8031
Yarn.resourcemanager.admin.address
192.168.0.194:8033
Yarn.resourcemanager.webapp.address
192.168.0.194:8088
Yarn.nodemanager.resource.memory-mb
2048
Note:
2048 means to set the memory size. More than 2 gigabytes is recommended, otherwise the interface can be jammed in the analysis.
6. Modify the configuration hadoop-env.sh file
[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/hadoop-env.sh
Modify the following:
Will: export JAVA_HOME=$ {JAVA_HOME}
Changed to: export JAVA_HOME=/usr/local/jdk1.7.0_79
7. Modify the configuration yarn-env.sh file
[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/yarn-env.sh
Add the following before the first if:
Export JAVA_HOME=/usr/local/jdk1.7.0_79
8. Modify the configuration slaves file
[root@master hadoop] # vim / usr/local/hadoop/etc/hadoop/slaves
Add all the IP of slave into, for example:
192.168.0.2
192.168.0.3
.
9. Synchronize profile
[root@master hadoop] # rsync-av / usr/local/hadoop/etc/ slave1:/usr/local/hadoop/etc
[root@master hadoop] # rsync-av / usr/local/hadoop/etc/ slave2:/usr/local/hadoop/etc
[root@master hadoop] # rsync-av / usr/local/hadoop/etc/ slave3:/usr/local/hadoop/etc
[root@master hadoop] # rsync-av / usr/local/hadoop/etc/ slave4:/usr/local/hadoop/etc
[root@master hadoop] # rsync-av / usr/local/hadoop/etc/ slave5:/usr/local/hadoop/etc
[root@master hadoop] # rsync-av / usr/local/hadoop/etc/ slave6:/usr/local/hadoop/etc
IV. Start the service
1. Initialization
[root@master hadoop] # / usr/local/hadoop/bin/hdfs namenode-format
[root@master hadoop] # echo $?
0
Note: 0 means there is no error. If the natural number is greater than 0, it indicates the initialization error. See the output to solve the problem.
two。 Start the service (yarn)
[root@master hadoop] # / usr/local/hadoop/sbin/start-yarn.sh
Starting yarn daemons
Starting resourcemanager, logging to / usr/local/hadoop/logs/yarn-root-resourcemanager-master.out
192.168.0.175: starting nodemanager, logging to / usr/local/hadoop/logs/yarn-root-nodemanager-slave1.out
192.168.0.191: starting nodemanager, logging to / usr/local/hadoop/logs/yarn-root-nodemanager-slave6.out
192.168.0.176: starting nodemanager, logging to / usr/local/hadoop/logs/yarn-root-nodemanager-slave2.out
192.168.0.184: starting nodemanager, logging to / usr/local/hadoop/logs/yarn-root-nodemanager-slave5.out
192.168.0.178: starting nodemanager, logging to / usr/local/hadoop/logs/yarn-root-nodemanager-slave3.out
192.168.0.183: starting nodemanager, logging to / usr/local/hadoop/logs/yarn-root-nodemanager-slave4.out
Note: version 2.7.1 starts the service with a script called start-all.sh (2.7.2 has been deprecated)
3. Stop service (yarn)
[root@master hadoop] # / usr/local/hadoop/sbin/stop-yarn.sh
4. Start the service (hdfs)
[root@master ~] # / usr/local/hadoop/sbin/start-dfs.sh
5. Stop service (hdfs)
[root@master ~] # / usr/local/hadoop/sbin/stop-dfs.sh
Check on 6.slave to see if hadoop is started
[root@slave1 ~] # ps aux | grep java
[root@slave1 ~] # netstat-nlp | grep java if there is no yum-y install net-tools
5. Browser access
Http://192.168.0.194:8088
Http://192.168.0.194:50070
5. Testing
1. Create a directory
[root@master ~] # cd / usr/local/hadoop/
[root@master hadoop] # bin/hdfs dfs-mkdir / 123
two。 View the catalog
[root@master hadoop] # bin/hdfs dfs-ls /
3. Copy test files
[root@master hadoop] # bin/hdfs dfs-copyFromLocal. / LICENSE.txt / 123
4. Analyze files (statistics files with more words and fewer words)
[root@master hadoop] # bin/hadoop jar. / share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount / 123/LICENSE.txt / 111/123
/ 111cus123: indicates the directory where the analysis results are stored
Jar: indicates what type of file to use
2.7.2.jar: indicates the specific file used, this is the best TAB completion, different versions have different names.
5. View analysis results
[root@master hadoop] # bin/hdfs dfs-cat / 111/123/part-r-00000
6. Delete directory
[root@master hadoop] # bin/hdfs dfs-rm-r / 123
VI. Expand the content
1. Start and stop a single node (hdfs)
Start Namenode:
$HADOOP_PREFIX/sbin/hadoop-daemon.sh-- config $HADOOP_CONF_DIR-- script hdfs start namenode
Stop Namenode:
$HADOOP_PREFIX/sbin/hadoop-daemon.sh-- config $HADOOP_CONF_DIR-- script hdfs stop namenode
Start Datanode:
$HADOOP_PREFIX/sbin/hadoop-daemons.sh-- config $HADOOP_CONF_DIR-- script hdfs start datanode
Stop Datanode:
HADOOP_PREFIX/sbin/hadoop-daemons.sh-config $HADOOP_CONF_DIR-script hdfs stop datanode
two。 Start and stop a single node (yarn)
Start ResourceManager:
$HADOOP_YARN_HOME/sbin/yarn-daemon.sh-- config $HADOOP_CONF_DIR start resourcemanager
Stop ResourceManager:
$HADOOP_YARN_HOME/sbin/yarn-daemon.sh-- config $HADOOP_CONF_DIR stop resourcemanager
Start NodeManager:
$HADOOP_YARN_HOME/sbin/yarn-daemons.sh-- config $HADOOP_CONF_DIR start nodemanager
Stop NodeManager:
$HADOOP_YARN_HOME/sbin/yarn-daemons.sh-- config $HADOOP_CONF_DIR stop nodemanager
Start WebAppProxy:
$HADOOP_YARN_HOME/sbin/yarn-daemon.sh-- config $HADOOP_CONF_DIR start proxyserver
Stop WebAppProxy:
$HADOOP_YARN_HOME/sbin/yarn-daemon.sh-- config $HADOOP_CONF_DIR stop proxyserver
3. reference
Official website 2.7.2 document URL: http://hadoop.apache.org/docs/stable/
