Namenode ha handover optimization

I. background

Currently, namenode uses the deployment mode of ha, but the automatic switching of ha often occurs in the system (the namenode node is actually normal). After investigation, it is found that the possible reasons are as follows:

The rpc port of HealthMonitor check's local namenode timed out, causing HealthMonitor to think that namenode hung up.

The session timeout on zk causes the currently held active lock (temp node) to be lost, causing automatic switching.

II. Optimization

The following optimization will adjust the corresponding timeout parameters for 1) and 2) to see if it works. Modify core-site.xml

Ha.health-monitor.rpc-timeout.ms 300000 ha.zookeeper.session-timeout.ms 180000 III. Operational impact

None. There is no need to restart hadoop, which will not cause namenode switching.

IV. Operation steps

The following operations assume master1 (active) and master2 (standby). Only by operating in this order can namenode not be switched.

1) Log in to master2 (standby), comment out the restart-zkfc.sh task in crontab, and execute

Sbin/hadoop-daemon.sh stop zkfc

2) Log in to master1 (active), comment out the restart-zkfc.sh task in crotab, and execute

Sbin/hadoop-daemon.sh stop zkfc

3) back up the core-site.xml files of master1 and master2, and add the above configuration items to the core-site.xml configuration files of master1 and master2

4) Log in to master1 (active) and execute

Sbin/hadoop-daemon.sh start zkfc

5) Log in to master2 (standby) and execute

Sbin/hadoop-daemon.sh start zkfc

6) check hadoop cluster status

7) important! Log in to master1 and master2 to restore the restart-zkfc.sh task in crontab

8) complete

Matters needing attention

1) you need to comment out the crontab task restart-zkfc.sh on the machine first, otherwise the program will automatically pull up the zkfc process. Cause namenode active to switch

2) restore the commented-out restart-zkfc.sh task in crontab after the task is completed