Hadoop goes deep into practical classics-lecture 02

Source: Tongtong Learning_Knowledge Learning and Sharing Platform

Reprint link: www.tongtongxue.com/archives/220.html

The Core of Hadoop

HDFS: Hadoop Distributed File System

MapReduce: Parallel Computing Framework

Yarn: Cluster Resource Management and Scheduling Framework

What is Hadoop?

Distributed Storage, Computing, Resource Management Platform for Big Data

Written by Doug Cutting

Inspired by three Google papers

Hadoop ecosystem

HDFS

master-slave structure

There is only one master node: namenode.

There are many: datanodes

namenode is responsible for:

Receive user operation request

Maintain directory structure of file system

Manage the relationship between files and blocks, between blocks and datanodes

DataNode is responsible for:

storage file

Files are stored in blocks on disk

To keep your data safe, there are multiple copies of your file.

Yarn

MapReduce under Yarn

master-slave structure

Master node, only one: ResourceManager

Control nodes, each Job has an MRAppMaster

From the node, there are many: YarnChild

ResourceManager is responsible for:

Receive computing tasks submitted by customers

Assign jobs to MRAppMaster for execution

Monitor MRAppMaster performance

MRAppMaster is responsible for:

Responsible for scheduling tasks performed by a Job

Assign Job to YarnChild Execution

Monitor YarnChild implementation