How to solve the problem of small Files in HDFS

Editor to share with you HDFS how to solve the problem of small files, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to understand it!

1. Understanding: I understand that a small file is a file whose data size stored in a block is much smaller than the storage capacity of the block. the existence of multiple small files consumes the same NameNode performance as the same number of large files, because NN needs to jump from one DN to another when dealing with multiple small files, but if it is dealing with several large files of the same size as multiple small files, DN efficiency and access speed will be greatly improved So we need to solve the problem of small files.

two。 Solution: Hadoop currently does not have a general system-level solution to the HDFS small file problem. Its three solutions, including Hadoop Archive,Sequence file and CombineFileInputFormat, require users to write programs to solve small file problems according to their own needs.

The above is all the contents of the article "how to solve the problem of small files in HDFS". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!