Hbase on Yarn- compilation and deployment of slider
1. Compile Slider
[root@plat1 apache-slider-0.91.0-incubating] # pwd/usr/local/src/apache-slider-0.91.0-incubating
Modify the pom file in this directory due to CDH version hadoop (2.6.0-cdh6.9.0), hbase (1.2.0-cdh6.9.0)
2.6.0-cdh6.9.01.2.0-cdh6.9.01.7.0
Comment out the dependency on hadoop-minicluster packages in slider-core and slider-funtest
Skip the maven test and execute the compile command
[root@plat1 apache-slider-0.91.0-incubating] # mvn clean package-Dmaven.test.skip=true-DskipTests
[INFO] Slider. SUCCESS [0.762 s]
[INFO] Command Logger Sample Application. SUCCESS [0.971 s]
[INFO] Slider Command Logger Slider App Package. SUCCESS [0.766 s]
[INFO] Slider Core.. SUCCESS [10.766 s]
[INFO] Slider Agent.. SUCCESS [1.245 s]
[INFO] Apache Slider Binary Distribution. SUCCESS [21.933 s]
[INFO] Slider Functional Tests.. SUCCESS [5.285 s]
[INFO] Slider App Packages.. SUCCESS [0.012 s]
[INFO] Slider Accumulo App Package.... SUCCESS [5.578 s]
[INFO] Slider HBase App Package.. SUCCESS [0.017 s]
[INFO] Slider Kafka App Package.. SUCCESS [1.046 s]
[INFO] Slider Storm App Package.. SUCCESS [0.011 s]
[INFO] Slider Tomcat App Package Testing WAR file. SUCCESS [0.886 s]
[INFO] Slider Tomcat App Package.. SUCCESS [1.666 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 51.578 s
[INFO] Finished at: 2016-12-01T12:00:14+08:00
[INFO] Final Memory: 89M/2916M
[INFO]
2. Download hbase-1.2.0-cdh6.9.0.tar.gz and store the directory / usr/local/src
Wget http://archive.cloudera.com/cdh6/cdh/5/hbase-1.2.0-cdh6.9.0.tar.gz
3. Generate slider-hbase-app-package-1.2.0-cdh6.9.0.zip
Mvn clean package-P hbase-app-package-D pkg.version=1.2.0-cdh6.9.0-Dpkg.name=hbase-1.2.0-cdh6.9.0.tar.gz-D pkg.src=/usr/local/src
-Dhbase.version= Hbase version
-Dpkg.version= application version
-Dpkg.name= app tar package file name
-folder where the Dpkg.src= app tar package file is located
4. Deploy the package that compiles slider successfully and configure the environment variables.
The successfully compiled package is located in SLIDER_SRC_HOME/slider-assembly/target/slider-0.91.0-incubating-all.tar.gz
Extract to / usr/local/slider
And configure the environment variable export SLIDER_HOME=/usr/local/slider
5. Create package
Slider package--install-name HBASE-package slider-0.91.0-incubating/slider-hbase-app-package-1.2.0-cdh6.9.0.zip
Verify:
[hadoop@nn1 slider] $hadoop fs-ls / user/hadoop/.slider/package/HBASE/slider-hbase-app-package-1.2.0-cdh6.9.0.zip-rw-r--r-- 3 hadoop hadoop 241347310 2016-12-01 13:43 / user/hadoop/.slider/package/HBASE/slider-hbase-app-package-1.2.0-cdh6.9.0.zip
6. Extract appConfig-default.json and resource-default.json from slider-hbase-app-package-1.2.0-cdh6.9.0.zip
Unzip slider-hbase-app-package-1.2.0-cdh6.9.0.zip appConfig-default.json-d / usr/local/slieder/
Unzip slider-hbase-app-package-1.2.0-cdh6.9.0.zip resource-default.json-d / usr/local/slider/
7. Modify appConfig-default.json
{"schema": "http://example.org/specification/v2.0.0","metadata": {}," global ": {" application.def ":" .slider / package/HBASE/slider-hbase-app-package-1.2.0-cdh6.9.0.zip "," create.default.zookeeper.node ":" true "," java_home ":" / usr/local/jdk "," system_configs ":" core-site " "site.global.app_user": "hadoop", "site.global.app_version": "1.2.0-cdh6.9.0", "site.global.app_root": "${AGENT_WORK_ROOT} / app/install/hbase-1.2.0-cdh6.9.0", "site.global.metric_collector_host": "${NN_HOST}", "site.global.metric_collector_port": "6188" "site.global.metric_collector_lib": "," site.global.hbase_instance_name ":" instancename "," site.global.hbase_root_password ":" secret "," site.global.user_group ":" hadoop "," site.global.monitor_protocol ":" http "," site.global.hbase_thrift_port ":" ${HBASE_THRIFT.ALLOCATED_PORT} "," site.global.hbase_thrift2_port ":" ${HBASE_THRIFT2.ALLOCATED_PORT} " "site.global.hbase_rest_port": "${HBASE_REST.ALLOCATED_PORT}", "site.hbase-env.hbase_master_heapsize": "1024m", "site.hbase-env.hbase_regionserver_heapsize": "1024m", "site.hbase-site.hbase.rootdir": "${DEFAULT_DATA_DIR} / data", "site.hbase-site.hbase.superuser": "${USER_NAME}" "site.hbase-site.hbase.tmp.dir": "${AGENT_WORK_ROOT} / work/app/tmp", "site.hbase-site.hbase.local.dir": "${hbase.tmp.dir} / local", "site.hbase-site.hbase.zookeeper.quorum": "${ZK_HOST}", "site.hbase-site.zookeeper.znode.parent": "${DEFAULT_ZK_PATH}", "site.hbase-site.hbase.regionserver.info.port": "0" "site.hbase-site.hbase.bulkload.staging.dir": "/ user/$ {USER_NAME} / hbase-staging", "site.hbase-site.hbase.coprocessor.region.classes": "org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint", "site.hbase-site.hbase.master.info.port": "${HBASE_MASTER.ALLOCATED_PORT}", "site.hbase-site.hbase.regionserver.port": "0" "site.hbase-site.hbase.master.port": "0"}, "components": {"slider-appmaster": {"jvm.heapsize": "1024m"}
8. Modify resource-default.json
{"schema": "http://example.org/specification/v2.0.0","metadata": {}," global ": {" yarn.log.include.patterns ":", "yarn.log.exclude.patterns": "", "yarn.component.instances": "1", "yarn.vcores": "1"}, "components": {"slider-appmaster": {"yarn.memory": "10240"} "HBASE_MASTER": {"yarn.role.priority": "1", "yarn.component.instances": "1", "yarn.placement.escalate.seconds": "10", "yarn.memory": "15000"}, "HBASE_REGIONSERVER": {"yarn.role.priority": "2", "yarn.component.instances": "1", "yarn.memory": "15000", "yarn.container.failure.threshold": "15" Yarn.placement.escalate.seconds: "60"}, "HBASE_REST": {"yarn.role.priority": "3", "yarn.component.instances": "1", "yarn.component.placement.policy": "1", "yarn.memory": "5560"}, "HBASE_THRIFT": {"yarn.role.priority": "4", "yarn.component.instances": "1", "yarn.component.placement.policy": "1" "yarn.memory": "5560"}, "HBASE_THRIFT2": {"yarn.role.priority": "5", "yarn.component.instances": "1", "yarn.component.placement.policy": "1", "yarn.memory": "5560"}
9. Start the hbase application on yarn
[hadoop@nn1 slider] $slider create hbase1-- template. / appConfig-default.json-- resources. / resources-default.json
