Overview and detailed explanation of HBase load balancing algorithm

The following gives you an overview and detailed explanation of HBase load balancing algorithm, hoping to give you some help in practical application. Load balancing involves more things, not much theory, and there are many books on the Internet. Today, we will use the accumulated experience in the industry to do an answer.

1. Summary

After version 0.92, the equalization algorithm of HBase can be customized through hbase.master.loadbalancer.class that implements the LoadBalancer interface.

two。 Load balancing implementation conditions.

Load balancing is performed at specific intervals (5 minutes by default for hbase.balancer.period).

Load balancing is not performed when the following scenarios are encountered:

1. The load balancing switch balanceSwitch is turned off.

2.HMase completes initialization.

There is an unprocessed regions in 3.RIT.

4. There is a DeadRegionserver being processed.

3. Specific algorithm of default load balancing

Sort all RegionServer by load, or by serverName if the load is the same.

Example:

RS1 11

RS6 2RS2 10

RS5 5RS3 9 is sorted as: RS3 9RS4 10

RS2 10RS5 5

RS4 10RS6 2

RS1 11

3.2 calculate whether equalization is currently needed by the following algorithm:

/ / Check if we even need to do any load balancingfloat average = (float) numRegions / numServers; / / for logging// HBASE-3681 check sloppiness firstint floor = (int) Math.floor (average * (1-slop)); int ceiling = (int) Math.ceil (average * (1 + slop)); if (serversByLoad.lastKey (). GetLoad () = floor) {/ / Skipped because no server outside (min,max) range LOG.info ("Skipping load balancing because balanced cluster "+" servers= "+ numServers +"+" regions= "+ numRegions +" average= "+ average +" + "mostloaded=" + serversByLoad.lastKey () .getLoad () + "leastloaded=" + serversByLoad.firstKey () .getLoad (); return null;}

According to the algorithm above, in this example:

Float average = 47 / 6 = 7.8

Int floor = (int) Math.floor (average * (1-slop)) = 6

Intceiling = (int) Math.ceil (average * (1 + slop)) = 9

Where slop is hbase.regions.slop and default is 0.2

You can see that there are only two region < floor on the RS6, so this example requires load balancing.

3.3.calculate min and max to get the minimum and maximum load

Int min = numRegions / numServers;int max = numRegions% numServers = 0? Min: min + 1

In this example:

Min = 7

Max = 8

3.4.The RS that is larger than the max load is reduced to the max load, and the removed Region is put into the regionsToMove queue for allocation.

RS6 2

RS6 2RS5 5

RS5 5RS3 9max:8RS3 8RS2 10

RS2 8RS4 10

RS4 8RS1 11

RS1 8

RegionsToMove

RS3 1

RS2 2

RS4 2

RS1 3

3.5 remove all RS less than min load from regionsToMove queue and add it to min load

RS6 2

RS6 7RS5 5

RS5 7RS3 8min:7RS3 8RS2 8

RS2 8RS4 8

RS4 8RS1 8

RS1 8

RegionsToMove

RS1 1

3.6 there are two situations here:

a. When all < min load RS rises to min load, when the regionsToMove queue has remaining region, the RS of traversing the min load starts to reach max.

b. When there is still RS load < min, get the required number of region from the max load RS (these region are just enough to enable the non-min RS load to min) into the regionsToMove queue, in the region allocation, so that