Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

What does big data mainly study?

Shulou Source: shulou.com Published: 2022-06-03 08:16:44 11月23日 Update

The basis of learning big data

1. Java SE, EE (SSM)

90% of big data's frames are written by Java.

2 、 MySQL

SQL on Hadoop

3 、 Linux

Big data's framework is installed on the Linux operating system

What do you need to learn

. In the process of getting started learning big data, I have encountered learning, industry, lack of systematic learning route, systematic learning planning, welcome you to join my big data learning communication skirt: 251956502, skirt files have my big data learning manual, development tools, PDF documents and books, you can download them by yourself.

Big data offline analysis

General processing of Thum1 data (T: maybe 1 day, 1 week, 1 month, 1 year)

A, Hadoop: generally do not choose the latest version, stepping on the pit is difficult to solve

(common, HDES, MapReduce, YARN)

The idea of building environment and processing data

B, Hive: big data's data warehouse

Manipulate data by writing SQL, similar to sql in MySQL database

C, HBase: NOSQL database based on HDFS

Column-oriented storage

D. Collaboration framework:

Sqoop (Bridge: HDFS "=" RDBMS)

Flume: collect information from log files

E. Scheduling framework

Anzkaban

Understand: crotab (included with Linux)

Zeus (Alibaba)

Oozie (cloudera)

F, Frontier Framework extension:

Kylin, impala, ElasticSearch (ES)

Big data real-time analysis

Mainly based on spark framework

Scala:OOP (object-oriented programming) + FP (function is programming)

SparkCore: analogical MapReduce

SparkSQL: analogical hive

SparkStreaming: real-time data processing

Kafka: message queuing

Frontier Framework extension: flink

Alibaba: blink

Big data machine learning

Spark MLlib: machine Learning Library

Pyspark programming: the combination of Python and spark

Recommendation system

Python data analysis

Python machine learning

Tags: Data learning framework systems machines analysis processing large numbers databases files programs programming design Baba operating systems books warehouses information functions basics Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno MariaDB Xiaomi Linux NVidia Shulou Tech Info