Hadoop+spark+scala environment-single instance version
Hadoop+spark environment-single instance version
1. Modify hostname and relational mapping
2. Turn off the firewall and create a folder
Mkdir / hadoop/tmp
Mkdir / hadoop/dfs/name
Mkdir / hadoop/dfs/data
Mkdir / hadoop/var
3. Configure the Scala environment
[root@hadoop conf] # vim / etc/profile
Export SCALA_HOME=/opt/scala2.11.12
Export PATH=.:$ {JAVA_HOME} / bin:$ {SCALA_HOME} / bin:$PATH
[root@hadoop conf] # scala-version / / check whether the installation is successful
4. Configure Spark environment / usr/java/jdk1.8.0_201-amd64
[root@hadoop conf] # vim / etc/profile
Export SPARK_HOME=/opt/spark2.2.3
Export PATH=.:$ {JAVA_HOME} / bin:$ {SCALA_HOME} / bin:$ {SPARK_HOME} / bin:$PATH
[root@hadoop conf] # vim / opt/spark2.2.3/conf/spark-env.sh
Export SCALA_HOME=/opt/scala2.11.12
Export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/
Export HADOOP_HOME=/opt/hadoop2.7.6
Export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
Export SPARK_HOME=/opt/spark2.2.3
Export SPARK_MASTER_IP=hadoop.master / / Host name
Export SPARK_EXECUTOR_MEMORY=1G / / set running memory
5. Configure the Hadoop environment
[root@hadoop hadoop2.7.6] # vim / etc/profile
Export HADOOP_HOME=/opt/hadoop2.7.6
Export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
Export HADOOP_OPTS= "- Djava.library.path=$HADOOP_HOME/lib"
Export PATH=$ {HADOOP_HOME} / bin
5.1Modification of cor-site.xml files
[root@hadoop hadoop] # vim core-site.xml
Hadoop.tmp.dir
/ hadoop/tmpAbase for other temporary directories.
Fs.default.name
Hdfs://hadoop.master:9000
5.2 modify hadoop-env.sh file / usr/java/jdk1.8.0_201-amd64
[root@hadoop hadoop] # vim hadoop-env.sh
Export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/
5.3 modify the hdfs-site.xml file
[root@hadoop hadoop] # vim hdfs-site.xml
Dfs.name.dir
/ hadoop/dfs/name
Path on the local filesystem where theNameNode stores the namespace and transactions logs persistently.
Dfs.data.dir
/ hadoop/dfs/data
Comma separated list of paths on the localfilesystem of a DataNode where it should store its blocks.
Dfs.replication
two
Dfs.permissions
False
Need not permissions
5.4 modify the mapred-site.xml file
[root@hadoop hadoop] # vim mapred-site.xml
Mapred.job.tracker
Hadoop.master:9001
Mapred.local.dir
/ hadoop/var
Mapreduce.framework.name
Yarn
6. Start Hadoop
Cd / opt/hadoop2.7.6/bin
. / hadoop namenode-format
Cd / opt/hadoop2.7.6/sbin
Start hdfs and yarn
Start-dfs.sh
Start-yarn.sh
Check and verify that it is normal
Enter 192.168.47.45purl 8088 and 192.168.47.45purl 50070 in the browser interface to see if you can access it.
7. Start Spark
Cd / opt/spark2.2.3/sbin
Start-all.sh
Http://192.168.47.45:8080/