20180524 Morning lesson record 17-Hadoop
1.MapReduce, tell me about your understanding.
Perform engine calculation
What is 2.Map?
Map: mapping function
What is 3.Reduce?
Reduce: specification function
4.shuffle, tell me about your understanding.
It is equivalent to shuffling cards and grouping according to key
Do you remember the picture of 5.wordcount?
Not too deep.
6. What does input slicing usually have to do with? You can have as many map task as you can with as many pieces as you want. Is that right?
Small files will be merged if the block size is incorrect.
Which abstract concept does the application AM and task run in 7.mapreduce architecture design?
Container
8. If the hdfs files are all small files, then there will be multiple map task. When we produce the general design, we need to merge or design our file storage to keep a file in a block size as far as possible (understand the problem)
Right
What are the memory parameters of the 9.NM node and the container?
Yarn.nodemanager.resource.memory-mb
Yarn.scheduler.minimum-allocation-mb
Yarn.scheduler.maximum-allocation-mb
Yarn.scheduler.minimum-allocation-vcores
Yarn.scheduler.maximum-allocation-vcores
10. Does the container use physical core or virtual core
Virtual core
11. Summarize the concept of mapreduce architecture design wordcount blog container, memory and vcore parameter settings