Shulou(Shulou.com)06/03 Report--

The following configuration is required for viewing SPARK history tasks:

Modify spark-defaults.conf configuration file

Spark.eventLog.enabled true

Spark.eventLog.dir hdfs://192.168.9.110:9000/eventLogs

Spark.eventLog.compress true

Modify spark-env.sh configuration file

Export SPARK_HISTORY_OPTS= "- Dspark.history.ui.port=18080-Dspark.history.retainedApplications=3-Dspark.history.fs.logDirectory=hdfs://DEV-HADOOP-01:9000/sparkHistoryLogs"

Both spark.eventLog.dir and SPARK_HISTORY_OPTS use hdfs to store log information for historical tasks in the above two configurations. So you need to create these two directories hdfs://192.168.9.110:9000/eventLogs and hdfs://DEV-HADOOP-01:9000/sparkHistoryLogs in hdfs ahead of time.

Enter the bin directory command of hadoop: cd / data/server/hadoop-2.6.0/bin

New hdfs://192.168.9.110:9000/eventLogs command:. / hadoop dfs-mkdir / eventLogs

New hdfs://DEV-HADOOP-01:9000/sparkHistoryLogs command:. / hadoop dfs-mkdir / sparkHistoryLogs

Restart the spark cluster

Start the history task process

Sbin/start-history-server.sh

The history task process initiates verification

Execute the jps command and the HistoryServer process appears

View Historical Task Web UI

Http://192.168.9.110:18080/

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.