How to understand big data Workflow Oozie

The content of this article mainly focuses on how to understand big data workflow Oozie. The content of the article is clear and clear. It is very suitable for beginners to learn and is worth reading. Interested friends can follow the editor to read together. I hope you can get something through this article!

Because the work needs to use oozie, but the information on the Internet more and more confused, after great efforts, finally clear, here, make a summary to help newcomers better get started, of course, vulgar words, inevitably omitted, welcome to exchange correction.

Provide scheduling and management of Hadoop tasks, which can manage not only MapReduce tasks, but also pig, hive, sqoop, spark and other tasks. Oozie is a workflow engine based on hadoop.

Two main components:

Workflow definition components: a list of Action (Action is a task node, eg:MapReduce task, pig task, hive task, etc.)

Scheduler component: schedulable WorkFlow (workflow is a task graph that defines a DAG, and the scheduler can decide to execute DAG (directed acyclic graph Direct Acyclic Graph) task graph at a certain time or in accordance with the conditions)

Two types of nodes in the Oozie workflow definition:

Control flow node: used to define logic judgment, eg:start, end, control flow execution path

Action node: user execution task node: hadoop task, oozie subprocess

Three files are required to define a complete Oozie workflow

1. Workflow.xml: (required) define workflow tasks (need to be put on HDFS)

2. Config-default: (optional) includes attribute values shared by all workflows

3. Job.properties: (required) attribute values for each workflow

Thank you for your reading. I believe you have a certain understanding of "how to understand big data Workflow Oozie". Go and practice it. If you want to know more about it, you can follow the website! The editor will continue to bring you better articles!