HDFS summary

Summary of HDFS's recent study

one。 The overall operation mechanism of hdfs:

Customers put a file into HDFS, in fact, HDFS will cut the file into blocks for storage, and will be stored separately in N Linux systems (the role of storing file blocks is: datanode) (the important thing is that the behavior of slicing is determined by the client) once the file is sliced and stored, HDFS must have a mechanism that needs to record the information of each file and the specific storage machine for each slicing. (the namenode that records the block information) needs to ensure the security of the data. HDFS stores multiple copies of each block in the cluster (the number of copies is determined by the client at the time of storage). Summary: a HDFS system consists of a server running namenode and N servers running datanode.

two。 Knowledge points:

Metadata is managed by namenode: the directory structure of HDFS and the block information of each file (id of blocks, number of copies, location, etc.)

three。 Namenode metadata record location:

Namenode stores real-time complete metadata in memory namenode also stores memory metadata on disk mirroring files at some time fsimagenamenode records (like adding) client operations that make metadata changes in edits log files.

four。 Secondarynamenode responsibility:

Secondarynamenode regularly downloads the fsimage image and the newly generated edits log from namenode, then loads the fsimage image into memory, then sequentially parses the edits file, modifies (integrates) the metadata objects in memory, serializes the memory metadata into a new fsimage, and uploads the fsimage image file to namenode. This process is called that every time checkpointsecondary namenode does checkpoint operation, the first time checkpoint needs to download fsimage, then you don't have to download it later, because you already have it on your own machine. Summary: secondarynamenode will download fsimage and edits to its own machine, serialize it into metadata and pass it to namenode, which can remember a single point of failure. So secondarynamenode had better be configured as well.