Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to design the HDFS architecture, which is very detailed and has a certain reference value. Friends who are interested must read it!

HDFS architecture design:

1.NN (NameNode): mainly maintains and manages the file namespace, including the file name, file permissions, creation time, number of copies, how many blocks the file is divided into, and which machines are quickly distributed (generate files fsimage and editlog).

two。 Understand the storage relationship between NN and blocks and the actual stored content under NN management.

Example: suppose NN is a square box with the capacity of 128000ML, each block is the same size square box with the capacity of 128ML, but there is only 3ml water in each box (the actual size of the file stored in each block).

How much ml is actually stored? 3 * (128000 3000ml) = storage here is managed by DN.

How many boxes are put in? 1000 (actually the number of blocks stored)

Overall, although only 3000ml is actually stored, it consumes the NN space of the total bytes of 1000 bottles' names, but if 1000 bottles are compressed to 500 or less (small file merging), you can consume less NN management space.

3.DN (DataNode): stores blocks, stores block checksums.

4.SN (SecondaryNode): do hourly inspection work checkpoint. (check the work of NN to generate files fsimage and editlog and package them into a new fsimage file to save.)

Figure understanding:

The above is all the content of this article "how to Design HDFS Architecture". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.