Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

20180524 Morning lesson record 17-Hadoop

Shulou Source: shulou.com Published: 2022-06-03 08:01:31 11月16日 Update

1.MapReduce, tell me about your understanding.

Perform engine calculation

What is 2.Map?

Map: mapping function

What is 3.Reduce?

Reduce: specification function

4.shuffle, tell me about your understanding.

It is equivalent to shuffling cards and grouping according to key

Do you remember the picture of 5.wordcount?

Not too deep.

6. What does input slicing usually have to do with? You can have as many map task as you can with as many pieces as you want. Is that right?

Small files will be merged if the block size is incorrect.

Which abstract concept does the application AM and task run in 7.mapreduce architecture design?

Container

8. If the hdfs files are all small files, then there will be multiple map task. When we produce the general design, we need to merge or design our file storage to keep a file in a block size as far as possible (understand the problem)

Right

What are the memory parameters of the 9.NM node and the container?

Yarn.nodemanager.resource.memory-mb

Yarn.scheduler.minimum-allocation-mb

Yarn.scheduler.maximum-allocation-mb

Yarn.scheduler.minimum-allocation-vcores

Yarn.scheduler.maximum-allocation-vcores

10. Does the container use physical core or virtual core

Virtual core

11. Summarize the concept of mapreduce architecture design wordcount blog container, memory and vcore parameter settings

Tags: Files designs memory containers no functions parameters size architecture concepts blogs impressions multiple applications engines yes physics programs brains nodes Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno Huawei vpn Microsoft Redmi NVidia