First acquaintance of Hadoop
Hadoop
Install the Ubuntu environment
192.168.1.64 HNClient
192.168.1.65 HNName
SUSE,Ubuntu vi cannot use the Backspace key to delete data
When deleting, press ESC and then X to delete the data
Insert data, using I
Open a new line under the current line, using o
Operate on HNClient
Norman@HNClient:~$ sudo vi / etc/hostname
Norman@HNClient:~$ HNClient
Norman@HNClient:~$ sudo apt-get install openssh-server
Norman@HNClient:~$ sudo vi / etc/hosts
192.168.1.64 HNClient
192.168.1.65 HNName
Norman@HNClient:~$ ssh-keygen (enter by default below)
Generating public/private rsa key pair.
Enter file in which to save the key (/ home/norman/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in / home/norman/.ssh/id_rsa.
Your public key has been saved in / home/norman/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:rj3kM5OeqxceqGP6DcofXa+hZFReLQmKqksqoYL+YH4 norman@HNClient
The key's randomart image is:
+-[RSA 2048]-+
|. | |
|. . . O |
|. . . +. | |
|. O. . | |
|. .. s |
|. O.o.oz. | |
| | + = o.roommates. | | |
| | Xo.E+ + X+ | |
| | oo.=+ |
+-[SHA256]-+
Norman@HNClient:~$ ssh localhost (ssh localhost, still requires password authentication)
Norman@localhost's password:
Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-36-generic i686)
Documentation: https://help.ubuntu.comManagement: https://landscape.canonical.comSupport: https://ubuntu.com/advantage
251 packages can be updated.
79 updates are security updates.
New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Wed Oct 31 23:14:08 2018 from 192.168.1.65
Norman@HNClient:~$ cat ~ / .ssh/id_rsa.pub > > ~ / .ssh/authorized_keys
Norman@HNClient:~$ ssh localhost (ssh localhost, no password authentication required)
Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-36-generic i686)
Documentation: https://help.ubuntu.comManagement: https://landscape.canonical.comSupport: https://ubuntu.com/advantage
251 packages can be updated.
79 updates are security updates.
New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Wed Oct 31 23:18:02 2018 from 127.0.0.1
Norman@HNClient:~$ ssh HNName (ssh HNName, still requires password authentication)
Norman@hnname's password:
Norman@HNClient:~$ ssh-copy-id-I / .ssh/id_rsa.pub norman@HNName
Norman@HNClient:~$ ssh HNName (ssh HNName, you can log in to HNName without a password)
Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-36-generic i686)
Documentation: https://help.ubuntu.comManagement: https://landscape.canonical.comSupport: https://ubuntu.com/advantage
254 packages can be updated.
79 updates are security updates.
New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Wed Oct 31 23:23:21 2018 from 192.168.1.64
Norman@HNName:~$
Operate on HNName
Norman@HNName:~$ sudo vi / etc/hosts
192.168.1.64 HNClient
192.168.1.65 HNName
Norman@HNName:~$ ssh-keygen (enter by default below)
Generating public/private rsa key pair.
Enter file in which to save the key (/ home/norman/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in / home/norman/.ssh/id_rsa.
Your public key has been saved in / home/norman/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:YXrPGdhKYkPsAroDlIZJ4sYdbrpHyvaMQccMV3GJn9I norman@HNName
The key's randomart image is:
+-[RSA 2048]-+
|. . Oo.. | |
|. + oo.. | |
| | oO.= = + | |
| | + .B. + E + |
| oo =. B S o |
| | +. = o = + o |
| o. . + |
|. * |
|. O |
+-[SHA256]-+
Norman@HNClient:~$ ssh localhost (ssh localhost, still requires password authentication)
Norman@localhost's password:
Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-36-generic i686)
Documentation: https://help.ubuntu.comManagement: https://landscape.canonical.comSupport: https://ubuntu.com/advantage
251 packages can be updated.
79 updates are security updates.
New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Wed Oct 31 22:55:29 2018 from 127.0.0.1
Norman@HNName:~$ cat ~ / .ssh/id_rsa.pub > > ~ / .ssh/authorized_keys
Norman@HNName:~$ ssh localhost (ssh localhost, no password authentication required)
Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-36-generic i686)
Documentation: https://help.ubuntu.comManagement: https://landscape.canonical.comSupport: https://ubuntu.com/advantage
254 packages can be updated.
79 updates are security updates.
New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Wed Oct 31 23:00:28 2018 from 127.0.0.1
Norman@HNName:~$ ssh-copy-id-I / .ssh/id_rsa.pub norman@hnclient
/ usr/bin/ssh-copy-id: INFO: Source of key (s) to be installed: "/ home/norman/.ssh/id_rsa.pub"
The authenticity of host 'hnclient (192.168.1.64)' can't be established.
ECDSA key fingerprint is SHA256:w5dwBrXor00JfFtpGXc0G/+deJJwmAxKmjXE32InhgA.
Are you sure you want to continue connecting (yes/no)? Yes
/ usr/bin/ssh-copy-id: INFO: attempting to log in with the new key (s), to filter out any that are already installed
/ usr/bin/ssh-copy-id: INFO: 1 key (s) remain to be installed-- if you are prompted now it is to install the new keys
Norman@hnclient's password:
Number of key (s) added: 1
Now try logging into the machine, with: "ssh 'norman@hnclient'"
And check to make sure that only the key (s) you wanted were added.
Norman@HNName:~$ ssh hnclient (ssh hnclient, you can log in to hnclient without a password)
Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-36-generic i686)
Documentation: https://help.ubuntu.comManagement: https://landscape.canonical.comSupport: https://ubuntu.com/advantage
251 packages can be updated.
79 updates are security updates.
New release '18.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Wed Oct 31 23:05:13 2018 from 192.168.1.58
Norman@HNClient:~$ exit
Norman@HNName:~$ sudo apt-get install openjdk-7-jdk
[sudo] password for norman:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package openjdk-7-jdk is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
Is only available from another source
E: Package 'openjdk-7-jdk' has no installation candidate
It is because the installation source of Ubuntu16.04 does not have openjdk7 by default, so you have to manually add the repository yourself, as follows:
Norman@HNName:~$ sudo add-apt-repository ppa:openjdk-r/ppa (add oracle openjdk ppa source) (add-apt-repository ppa: xxx/ppa) means to get the latest personal package archive source, add it to the current apt library, and automatically import the public key.)
Norman@HNName:~$ sudo apt-get update
Norman@HNName:~$ sudo apt-get install openjdk-7-jdk
Norman@HNName:~$ java-version
Java version "1.7.0Y95"
OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-3)
OpenJDK Client VM (build 24.95-b01, mixed mode, sharing)
Norman@HNName:~$ wget http://archive.apache.org/dist/hadoop/core/hadoop-1.2.0/hadoop-1.2.0-bin.tar.gz
Norman@HNName:~$ tar-zxvf hadoop-1.2.0-bin.tar.gz
Norman@HNName:~$ sudo cp-r hadoop-1.2.0 / usr/local/hadoop
Norman@HNName:~$ dir / usr/local/hadoop
Bin hadoop-ant-1.2.0.jar hadoop-tools-1.2.0.jar NOTICE.txt
Build.xml hadoop-client-1.2.0.jar ivy README.txt
C++ hadoop-core-1.2.0.jar ivy.xml sbin
CHANGES.txt hadoop-examples-1.2.0.jar lib share
Conf hadoop-minicluster-1.2.0.jar libexec src
Contrib hadoop-test-1.2.0.jar LICENSE.txt webapps
Norman@HNName:~$ sudo vi $HOME/.bashrc (add the following at the end)
Export HADOOP_PREFIX=/usr/local/hadoop
Export PATH=$PATH:$HADOOP_PREFIX/bin
Norman@HNName:~$ exec bash
Norman@HNName:~$ $PATH
Norman@HNName:~$ sudo vi / usr/local/hadoop/conf/hadoop-env.sh
(The java implementation to use. Required.)
Export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-i386
(Extra Java runtime options. Empty by default. Set to disable IPv6)
Export HADOOP_OPTS=-Djava.net.preferIP4Stack=true
Installing Apache Hadoop (Single Node)
Norman@HNName:~$ sudo vi / usr/local/hadoop/conf/core-site.xml
Fs.default.name
Hdfs://HNName:10001
Hadoop.tmp.dir
/ usr/local/hadoop/tmp
Norman@HNName:~$ sudo vi / usr/local/hadoop/conf/mapred-site.xml
Mapred.job.tracker
HNName:10002
Norman@HNName:~$ sudo mkdir / usr/local/hadoop/tmp
Norman@HNName:~$ sudo chown norman / usr/local/hadoop/tmp
Norman@HNName:~$ hadoop namenode-format (you can see the following description of success)
18-11-01 19:07:36 INFO common.Storage: Storage directory / usr/local/hadoop/tmp/dfs/name has been successfully formatted.
Norman@HNName:~$ hadoop-daemons.sh start namenode (with the following error)
Localhost: mkdir: cannot create directory? usr/local/hadoop/libexec/../logs? Permission denied
Localhost: chown: cannot access'/ usr/local/hadoop/libexec/../logs': No such file or directory
Localhost: starting namenode, logging to / usr/local/hadoop/libexec/../logs/hadoop-norman-namenode-HNName.out
Localhost: / usr/local/hadoop/bin/hadoop-daemon.sh: line 137: / usr/local/hadoop/libexec/../logs/hadoop-norman-namenode-HNName.out: No such file or directory
Localhost: head: cannot open'/ usr/local/hadoop/libexec/../logs/hadoop-norman-namenode-HNName.out' for reading: No such file or directory
Localhost: / usr/local/hadoop/bin/hadoop-daemon.sh: line 147: / usr/local/hadoop/libexec/../logs/hadoop-norman-namenode-HNName.out: No such file or directory
Localhost: / usr/local/hadoop/bin/hadoop-daemon.sh: line 148: / usr/local/hadoop/libexec/../logs/hadoop-norman-namenode-HNName.out: No such file or directory
Norman@HNName:~$ ll / usr/local
Total 44
Drwxr-xr-x 11 root root 4096 Nov 1 02:02. /
Drwxr-xr-x 11 root root 4096 Feb 28 2018.. /
Drwxr-xr-x 2 root root 4096 Feb 28 2018 bin/
Drwxr-xr-x 2 root root 4096 Feb 28 2018 etc/
Drwxr-xr-x 2 root root 4096 Feb 28 2018 games/
Drwxr-xr-x 15 root root 4096 Nov 1 20:05 hadoop/
Drwxr-xr-x 2 root root 4096 Feb 28 2018 include/
Drwxr-xr-x 4 root root 4096 Feb 28 2018 lib/
Lrwxrwxrwx 1 root root 9 Jul 26 23:29 man-> share/man/
Drwxr-xr-x 2 root root 4096 Feb 28 2018 sbin/
Drwxr-xr-x 8 root root 4096 Feb 28 2018 share/
Drwxr-xr-x 2 root root 4096 Feb 28 2018 src/
Norman@HNName:~$ sudo chown norman / usr/local/hadoop
Norman@HNName:~$ ll / usr/local
Total 44
Drwxr-xr-x 11 root root 4096 Nov 1 02:02. /
Drwxr-xr-x 11 root root 4096 Feb 28 2018.. /
Drwxr-xr-x 2 root root 4096 Feb 28 2018 bin/
Drwxr-xr-x 2 root root 4096 Feb 28 2018 etc/
Drwxr-xr-x 2 root root 4096 Feb 28 2018 games/
Drwxr-xr-x 15 norman root 4096 Nov 1 20:05 hadoop/
Drwxr-xr-x 2 root root 4096 Feb 28 2018 include/
Drwxr-xr-x 4 root root 4096 Feb 28 2018 lib/
Lrwxrwxrwx 1 root root 9 Jul 26 23:29 man-> share/man/
Drwxr-xr-x 2 root root 4096 Feb 28 2018 sbin/
Drwxr-xr-x 8 root root 4096 Feb 28 2018 share/
Drwxr-xr-x 2 root root 4096 Feb 28 2018 src/
Norman@HNName:~$ hadoop-daemons.sh start namenode
Localhost: starting namenode, logging to / usr/local/hadoop/libexec/../logs/hadoop-norman-namenode-HNName.out
Norman@HNName:~$ start-all.sh
Norman@HNName:~$ jps
23297 DataNode
23610 TaskTracker
23484 JobTracker
23739 Jps
23102 NameNode
23416 SecondaryNameNode
Norman@HNName:~$ dir / usr/local/hadoop/bin
Hadoop hadoop-daemon.sh rcc start-all.sh start-dfs.sh start-mapred.sh stop-balancer.sh stop-jobhistoryserver.sh task-controller
Hadoop-config.sh hadoop-daemons.sh slaves.sh start-balancer.sh start-jobhistoryserver.sh stop-all.sh stop-dfs.sh stop-mapred.sh
Http://192.168.1.65:50070/dfshealth.jsp
Http://192.168.1.65:50030/jobtracker.jsp
Http://192.168.1.65:50060/tasktracker.jsp
Managing HDFS
Http://www.gutenberg.org/files/2600/2600-0.txt (download text file)
Copy web page content to war_and_peace.txt
Https://www.ncdc.noaa.gov/orders/qclcd/ (download any data)
QCLCD201701.zip,QCLCD201702.zip, and then extract 201701hourly.txt, 201702hourly.txt
Operate on HNClient
Put the data war_and_peace.txt to / home/norman/data/book
Put the data 201701hourly.txtjol 201702hourly.txt to / home/norman/data/weather
Norman@HNClient:~$ sudo mkdir-p / home/norman/data/book
Norman@HNClient:~$ sudo mkdir-p / home/norman/data/weather
Norman@HNClient:~$ sudo chown norman/ home/norman/data/weather
Norman@HNClient:~$ sudo chown norman/ home/norman/data/book
Norman@HNClient:~$ sudo add-apt-repository ppa:openjdk-r/ppa
Norman@HNClient:~$ sudo apt-get update
Norman@HNClient:~$ sudo apt-get install openjdk-7-jdk
Norman@HNClient:~$ java-version
Norman@HNClient:~$ wget http://archive.apache.org/dist/hadoop/core/hadoop-1.2.0/hadoop-1.2.0-bin.tar.gz
Norman@HNClient:~$ tar-zxvf hadoop-1.2.0-bin.tar.gz
Norman@HNClient:~$ sudo cp-r hadoop-1.2.0 / usr/local/hadoop
Norman@HNClient:~$ sudo vi $HOME/.bashrc (add the following at the end)
Export HADOOP_PREFIX=/usr/local/hadoop
Export PATH=$PATH:$HADOOP_PREFIX/bin
Norman@HNClient:~$ exec bash
Norman@HNClient:~$ $PATH
Norman@HNClient:~$ sudo vi / usr/local/hadoop/conf/hadoop-env.sh
(The java implementation to use. Required.)
Export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-i386
(Extra Java runtime options. Empty by default. Set to disable IPv6)
Export HADOOP_OPTS=-Djava.net.preferIP4Stack=true
Norman@HNClient:~$ sudo vi / usr/local/hadoop/conf/core-site.xml
Fs.default.name
Hdfs://HNName:10001
Hadoop.tmp.dir
/ usr/local/hadoop/tmp
Norman@HNClient:~$ sudo vi / usr/local/hadoop/conf/mapred-site.xml
Mapred.job.tracker
HNName:10002
Norman@HNClient:~$ hadoop fs-mkdir test
Norman@HNClient:~$ hadoop fs-ls
Found 1 items
Drwxr-xr-x-norman supergroup 0 2018-11-02 01:17 / user/norman/test
Norman@HNClient:~$ hadoop fs-mkdir hdfs://hnname:10001/data/small
Norman@HNClient:~$ hadoop fs-mkdir hdfs://hnname:10001/data/big
Open the web page to http://192.168.1.65:50070
Http://192.168.1.65:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/
Norman@HNClient:~$ hadoop fs-rmr test (Test deletion)
Deleted hdfs://HNName:10001/user/norman/test
Norman@HNClient:~$ hadoop fs-moveFromLocal / home/norman/data/book/war_and_peace.txt hdfs://hnname:10001/data/small/war_and_peace.txt
You can see the following data
Norman@HNClient:~$ hadoop fs-copyToLocal hdfs://hnname:10001/data/small/war_and_peace.txt / home/norman/data/book/war_and_peace.bak.txt (test replicated locally)
Norman@HNClient:~$ hadoop fs-put / home/norman/data/weather hdfs://hnname:10001/data/big
You can see the following data
Norman@HNClient:~$ hadoop dfsadmin-report
Configured Capacity: 19033165824 (17.73 GB)
Present Capacity: 13114503168 (12.21 GB)
DFS Remaining: 12005150720 (11.18 GB)
DFS Used: 1109352448 (1.03GB)
DFS Used%: 8.46%
Under replicated blocks: 19
Blocks with corrupt replicas: 0
Missing blocks: 0
Datanodes available: 1 (1 total, 0 dead)
Name: 192.168.1.65:50010
Decommission Status: Normal
Configured Capacity: 19033165824 (17.73 GB)
DFS Used: 1109352448 (1.03GB)
Non DFS Used: 5918662656 (5.51 GB)
DFS Remaining: 12005150720 (11.18 GB)
DFS Used%: 5.83%
DFS Remaining%: 63.07%
Last contact: Fri Nov 02 01:49:43 GMT-08:00 2018
Norman@HNClient:~$ hadoop dfsadmin-safemode enter (safemode is required for upgrade)
Safe mode is ON
Norman@HNClient:~$ hadoop dfsadmin-safemode leave
Safe mode is OFF
Operate on HNName
Norman@HNName:~$ hadoop fsck-blocks
Status: HEALTHY
Total size: 1100586452 B
Total dirs: 13
Total files: 4
Total blocks (validated): 19 (avg. Block size 57925602 B)
Minimally replicated blocks: 19 (100.0)
Over-replicated blocks: 0 (0.0%)
Under-replicated blocks: 19 (100.0)
Mis-replicated blocks: 0 (0.0%)
Default replication factor: 3
Average block replication: 1.0
Corrupt blocks: 0
Missing replicas: 38 (200.0)
Number of data-nodes: 1
Number of racks: 1
FSCK ended at Fri Nov 02 01:54:46 GMT-08:00 2018 in 1049 milliseconds
The filesystem under path'/'is HEALTHY
Norman@HNName:~$ hadoop fsck / data/big
Status: HEALTHY
Total size: 1097339705 B
Total dirs: 2
Total files: 2
Total blocks (validated): 17 (avg. Block size 64549394 B)
Minimally replicated blocks: 17 (100.0)
Over-replicated blocks: 0 (0.0%)
Under-replicated blocks: 17 (100.0)
Mis-replicated blocks: 0 (0.0%)
Default replication factor: 3
Average block replication: 1.0
Corrupt blocks: 0
Missing replicas: 34 (200.0)
Number of data-nodes: 1
Number of racks: 1
FSCK ended at Fri Nov 02 19:33:55 GMT-08:00 2018 in 14 milliseconds
The filesystem under path'/ data/big' is HEALTHY
