What is the method of Storm stand-alone installation?

This article mainly explains the "Storm stand-alone installation method is what", the article explains the content is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "what is the method of Storm stand-alone installation"!

The key points are:

Conf.setNumWorkers (3);-> the processes of the created worker will be evenly distributed to the cluster nodes of the storm

StormSubmitter.submitTologyWithProgressBar (args [0], conf,builder.createTology ())

-> args [0]-- is the name of the last Blot in the previous code, count

1. Cluster members of Storm:

Nimbus

ZK

Supervisor

Worker (including executer)

Function:

Nimbus: responsible for receiving client data and scheduling. When scheduling, ask zk for supervisor information, and then schedule according to the information. Nimbus and supervisor do not communicate directly.

Supervisor: responsible for monitoring the running status of subordinate nodes. If supervisor dies, nimbus will reschedule according to the information returned by zk.

Worker:worker is a process. When worker gets the task, it starts an executor thread, and then the executor thread works.

Note: nimbus is responsible for task scheduling and receiving client data, supervisor and worker are processes, supervisor is the management process, worker is the worker process, and executor is the real worker thread.

ZK: highly available, configuration information is stored on znode

2. Installation of Strom

1) deployment depends on the environment-Java 6 + and Python 2.6.6 + are required

2) deploy zookeeper

-3.4.5 +

Why should zk use 3.4.5, because it supports snapshot of disk and regular deletion of namenode to avoid disk being full.

3), distribute the storm package-version 0.9.4 is used here

4), configure storm-modify storm.yaml configuration file

5) start strom

Stand-alone installation

Java,python deployment runs java-version and python-version validated versions

Upload apache-storm-0.9.5.tar.gz

Tar zxf apache-storm-0.9.5.tar.gz

Cd apache-storm-0.9.5

Mkdir logs

. / bin/storm dev-zookeeper > >. / logs/zk.out 2 > & 1 &

. / bin/storm nimbus > >. / logs/nimbus.out 2 > & 1 &

. / bin/storm ui > >. / logs/ui.out 2 > & 1 &

. / bin/storm supervisor > >. / logs/supervisor.out 2 > & 1 &

. / bin/storm logviewer > >. / logs/logviewer.out 2 > & 1 &

Authentication: login address http://node11:8080/ view

Jobs-> this command is used to view the tasks currently running in the background of the terminal.

Ss-tanl-> View port occupancy

Port 6627-"nimbus"

Supervisor-"No ports, just monitor worker

Execute this command and feel the storm

. / bin/storm jar examples/storm-starter/storm-starter-topologies-0.9.4.jar storm.starter.WordCountTopology wordcount

Deployment of storm three nodes

Deploy Zookeeper clusters with 3 nodes (no demonstration here)

Node11 is nimbus, start configuration

1. Decompress the storm package

2. Modify the storm/conf/storm.yaml configuration file as follows (note that the spaces in it are useful and cannot be deleted)

Storm.zookeeper.servers:

-"192.168.47.21"

-"192.168.47.22"

-"192.168.47.23"

Storm.local.dir: "/ tmp/storm"

Nimbus.host: "192.168.47.21"

Supervisor.slots.ports:

-6700

-6701

-6702

-6703

3. Create a logs directory in the storm directory

4. Start the ZooKeeper cluster

5. Start Nimbus on node11

$. / bin/storm nimbus > >. / logs/nimbus.out 2 > & 1 &

$tail-f logs/nimbus.log

$. / bin/storm ui > >. / logs/ui.out 2 > & 1 &

$tail-f logs/ui.log

6. Node nod12 and node13 start supervisor. According to configuration, there are 4 slots for each supervisor started.

$. / bin/storm supervisor > >. / logs/supervisor.out 2 > & 1 &

$tail-f logs/supervisor.log

Of course, node11 can also start supervisor.

7. Http://192.168.47.21:8080/

$bin/storm jar examples/storm-starter/storm-starter-topologies-0.9.4.jar storm.starter.WordCountTopology test

8. Environment variables may or may not be configured.

Export STORM_HOME=/usr/storm

Export PATH=$PATH:$STORM_HOME/bin

Thank you for your reading, the above is the content of "what is the method of Storm stand-alone installation". After the study of this article, I believe you have a deeper understanding of what the method of Storm stand-alone installation is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!