Summary of disk loss handling in development environment hadoop&hbase cluster environment
Problem description:
Cluster
192.168.22.178 master1
192.168.22.179 master2
192.168.22.40 data1&zk&kafka&es
192.168.22.69 data2&zk&kafka&es
192.168.22.177 data3&kafka&es
192.168.22.180 data4
Hosts 192.168.22.40 and 192.168.22.177 are not only the datanode of the hadoop cluster but also the regionserver of hbase. The data is stored in / data and / data2 directories. The disks mounted in the two directories are lost due to abnormal conditions, and the cluster is abnormal and cannot provide services normally.
Restore steps:
1. Wait for the CVM to resume its status, and then reapply for formatted mounting of the disk.
3. Rebuild zk, kafka and es clusters
2. Copy the directories under / data and / data1 on the host 192.168.22.69 to the two machines, and then delete the log files and data.
3. Synchronize journal data
Copy the / data/hadoop_data/journal/masters directory on host 192.168.22.69 to the / data/hadoop_data/journal directory of the other two machines.
4. Start two hadoop services and switch the hbase data storage directory
$vim / data/hbase/conf/hbase-site.xml
Hbase.rootdir
Hdfs://masters/hbase
> >
Hbase.rootdir
Hdfs://masters/hbase1
Synchronize configuration files to all cluster nodes
The above method is to completely abandon the hbase data, this is because it is a test environment, if it is not a test environment, if the data is more important, you need another way, such as the way shown on the following website:
Https://blog.csdn.net/anyking0520/article/details/79064967
Log in to the zk server 192.168.22.40 and delete the data information of the hbase table saved in the zk cluster
# cd / home/box/zookeeper-3.4.6/bin
#. / zkCli.sh
After entering shell, use the following command to delete
Rmr / hbase/table
5. Turn off hadoop cluster protection mode
On the primary node
$cd / data/hadoop/bin
. / hadoop dfsadmin-safemode leave
6. Pay attention to the order of starting the cluster
6.1. resume and start zookeeper cluster
On the primary node
6.2.Starting hadoop cluster service
After configuring the hadoop cluster, start the entire hadoop cluster service
$cd / data/hadoop/sbin
$. / start-all.sh
Leave the hadoop cluster security mode
Check whether the status of the hdfs cluster is normal
$cd / data/hadoop/bin
. / hdfs dfsadmin-report
6.3. turn off hadoop cluster protection mode
6.4.After hbase is configured, start the hbase cluster
$cd / data/hbase/bin
$start-hbase.sh
Check whether the status of the hbase cluster is normal
$cd / data/hbase/bin
$. / hbase shell
After entering shell, use the status command to check the status
