Directions for hbase optimization and planning

How to use hbase well, and how to define how to use hbase well?

To achieve the best performance (throughput, read-write delay) with the least system resources (CPU,IO, etc.) on the basis of ensuring system stability and availability is "good use".

The optimization direction is as follows:

(1) HDFS-related configuration optimization, (2) HBase server-side optimization (GC optimization, Compaction optimization, hardware configuration optimization), (3) column family design optimization, (4) client optimization, etc.

Hbase planning direction: divided into A.hbase memory planning, B.hbase cluster planning

The reference link for A.hbase memory planning is: hbase memory planning (read more, write less and write more, read less) https://blog.51cto.com/12445535/2373788

The reference link for B.hbase cluster planning is: hbase cluster planning (cluster business planning, cluster capacity planning, Region planning)

Https://blog.51cto.com/12445535/2375540

(4) the fourth point (4) client optimization has been mentioned earlier through timeout mechanism and retry mechanism. The reference link is:

HBase client Rpc retry mechanism and client parameter optimization. Https://blog.51cto.com/12445535/2373709

Parameter optimization practice of hbase client timeout mechanism: https://blog.51cto.com/12445535/2373731

(3) Summary of design optimization of column families.

Hbase column family design (largely determines the performance of reading and writing) / / reference link HBase best practices-column family design optimization http://hbasefly.com/2016/07/02/hbase-pracise-cfsetting/

Hbase create table statement

Create 'NewsClickFeedback', {NAME= >' Toutiao',VERSIONS= > 1meme BLOCKCACHE = > true,BLOOMFILTER= > 'ROW',COMPRESSION= >' SNAPPY',TTL = > '259200, DATA_BLOCK_ENCODING = >' PREFIX_TREE', BLOCKSIZE = > '65536'}, {SPLITS = > ['1', 'PREFIX_TREE', BLOCKSIZE = >' 65536'}, {SPLITS = > ['1','1','2','3','5','5','6','7','8','9',

Summary:

1. For random read-based services, the size of BlockSize can be appropriately reduced to achieve better read performance. The default is 64K

2. For scan-based services, we can appropriately increase the size of BlockSize to obtain better read performance.

[hint:

1. It can be seen that if the business request is mainly a Get request, you can consider setting the block size smaller.

two。 If you focus on Scan requests, you can increase the block size; the default 64K block size is a balance between Scan and Get.

]

3. Data Encoding / Compression Compress/DeCompress (Compression / decompression)

Snappy: generally speaking, Snappy has the lowest compression ratio, but the highest codec rate and the lowest consumption of CPU. At present, Snappy is generally recommended.

4. Encode/Decode (data encoding function)

Recommended: DATA_BLOCK_ENCODING = > 'PREFIX_TREE' / / this configuration is not recommended for production due to security considerations.

(2) gc optimization for hbase server-side optimization can be found in hbase gc series of blogs

Https://blog.51cto.com/12445535/category16.html

(2) compaction optimization of hbase server optimization

Https://blog.51cto.com/12445535/2375292

Https://blog.51cto.com/12445535/2375361

(2) hbase server-side optimized hbase uses hardware optimization (make good use of the operating system)

Https://blog.51cto.com/12445535/2375515

(1) Optimization related to hdfs

For the direction of optimization when there are more hbase reads, see in-depth discussion on hbase read performance Optimization (20190308) hdfs optimization in https://blog.51cto.com/12445535/2360206 (enable HDFS fast reading, is the data localization rate too low? )

Hbase RIT mechanism

(two very important cases: permanent RIT status case study (case 1: Compaction permanent blocking case 2: HDFS file exception))

Region-In-Trasition mechanism

/ / what is RIT Region-In-Transition?

It is the mechanism of Region transition, which actually refers to the transition of Region state in a specific operation.

Region exists in 15 states.

The four actions that trigger the state transition of Region are assign, unassign, split and merge [allocation de-allocation split merge], while many other operations can be split into unassign and assign. For example, the move operation actually starts with unassign and then assign.

Http://hbasefly.com/2016/09/08/hbase-rit/

Hbase multi-tenancy

(1) Resource restrictions, which are mainly limited to users, namespace, QPS and request size of tables. For more information, please see HBase-11598

(2) Resource scheduling, mainly for priority scheduling of tasks, usually giving priority to real-time interactive and small tasks, while batch operation tasks or long-time operation tasks (large scan) have relatively low priority. For more information, please see HBase-10993.

(3) Resource isolation. Distribute different tables to different RegionServer through physical isolation. For more information, please see HBase-6721.

Resource limits-Quotas command

Http://hbasefly.com/2016/09/26/hbase-mutiltenant-1/