Data processing flow

Flow chart analysis

A typical BI system flow is as follows:

Because it is dealing with huge amounts of data, the technology used in each part of the process is completely different from that of traditional BI. Subsequent courses will explain them one by one:

1) data acquisition: customize the development of acquisition programs, or use the open source framework FLUME

2) data preprocessing: custom development mapreduce programs run in Hadoop cluster

3) data Warehouse Technology: Hive based on hadoop

4) data export: hadoop-based sqoop data import and export tool

5) data visualization: customize the development of web programs or use products such as kettle

6) process scheduling of the whole process: oozie tools or other similar open source products in the hadoop ecosystem

Technical architecture diagram of the project