How should MapReduce be applied and developed

MapReduce should be how to apply the development, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

API for configuration

Configuration org.apache.hadoop.conf

Configure the development environment

Different environments use different configuration files, the same as the web development class

HADOOP_USER_NAME sets the hadoop user name

HADOOP_CONF_DIR configuration information environment variables

Auxiliary class Configured Tool ToolRunner

Writing unit tests with MRUnit

Run test data locally

Run a job on the local job runner

Test driver

Use the local job runner

Use a mini cluster to run it

Run on a cluster

Packing job

Distributed environment is packaged into JAR files

Start the job

Job debugging

By printing statements

The counter can measure the severity of the problem.

Journal

Remote debugging

Job optimization

Number of mapper

Number of reducer

Combiner

Compression of intermediate value

Custom sequence

Adjust shuffle

Workflow of MapReduce

It is usually to add more jobs, not to increase the complexity of the jobs.

For more complex problems, consider using Pig, hive, Cascading, Crunch or Spark

JobControl

Apache Oozie is a system that runs workflows, which consist of interdependent jobs

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.