Shulou(Shulou.com)06/01 Report--

This article mainly introduces "how to realize real-time streaming from RDBMS to Hadoop". In daily operation, I believe many people have doubts about how to realize real-time streaming from RDBMS to Hadoop. The editor consulted all kinds of data and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "how to realize real-time streaming from RDBMS to Hadoop". Next, please follow the editor to study!

Opportunities for Kafka to develop its talents: overall solution Architecture

The following figure shows that in the overall solution architecture, RDBMS's business data passed to the target Hive table combines Kafka, Flume, and Hive transaction functions.

7-step real-time stream transfer to Hadoop

Now going into the details of the solution, I'll show you how to stream data to Hadoop in a few simple steps.

1. Extract data from a relational database management system (RDBMS)

All relational databases have a log file that records recent transactions. The step of our transport streaming solution is to obtain these transactions in a format that can be transmitted to Hadoop. After talking about the extraction mechanism, you have to take up a separate blog post-so if you want to know more about this process, please contact us.

two。 Establish Kafka Producer

The process of publishing a message to a Kafka topic is called a "producer". A "topic" is a classified message saved by Kafka. The RDBMS deal will be turned into a Kafka topic. For this example, let's consider the database of the sales team, where the transactions are published as a Kafka theme. The following steps are required to establish a Kafka producer:

3. Set up Hive

Next, we will create a table in Hive that is ready to receive database transactions from the sales team. In this example, we will create a customer table:

In order for Hive to process transactions, the following settings are required in the configuration:

Hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.dbtxnmanager

4. Set up Flume Agent for streaming from Kafka to Hive

Now let's look at how to create a Flume proxy that takes data from a Kafka topic and sends it to the Hive table.

Follow the steps to set up the environment, and then set up the Flume agent:

Next, create a log4j properties file as follows:

Then use the following configuration file for the Flume agent:

5. Open the Flume proxy

Open the Flume agent using the following command:

$/ usr/hdp/apache-flume-1.6.0/bin/flume-ng agent-n flumeagent1-f ~ / streamingdemo/flume/conf/flumetohive.conf

6. Turn on Kafka Stream

The following example is a simulated transaction message that needs to be generated by the source database in a real system. For example, the following may come from the Oracle data stream of duplicate SQL transactions that have been submitted to the database or from GoledenGate.

7. Receive Hive data

All of the above is done, now send the data from Kafka, and you will see that within seconds, the data stream is sent to the Hive table.

At this point, the study on "how to realize the real-time streaming from RDBMS to Hadoop" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.