What is the implementation method of Datax offline data extraction?
This article mainly explains "what is the implementation method of Datax offline data extraction", interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "what is the implementation method of Datax offline data extraction"?
1. Download and install Datax software (jdk version 1.8 or above must be installed)
[root@localhost ~] # tar xvf jdk-8u65-linux-x64.tar.gz
[root@localhost ~] # mv jdk1.8.0_151 / usr/local/jdk1.8.0_151
[root@localhost ~] # vim / etc/profile
Export PATH=$PATH:/usr/local/jdk1.8.0_151/bin
[root@localhost ~] # vim / etc/ld.so.conf.d/mysql-x86_64.conf
/ usr/local/jdk1.8.0_151/lib
[root@localhost ~] # tar xvf datax.tar.gz
[root@localhost ~] # cd datax/job/
two。 Edit the configuration file (first synchronize the table structure with kettle)
[root@localhost job] # vim job1.json
{"job": {"setting": {"speed": {"channel": 5}}, "content": [{"reader": {"name": "oraclereader" "parameter": {"username": "upcenter", "password": "upcenter", "column": ["*"] "connection": [{"table": ["STOCK_CONC"] "jdbcUrl": ["jdbc:oracle:thin:@192.168.7.7:1521:upqc"]}, "writer": {"name": "mysqlwriter" "parameter": {"writeMode": "update", "username": "wangying", "password": "wangying", "column": ["*"] "connection": [{"jdbcUrl": "jdbc:mysql://172.16.8.93:3306/db_stktag" "table": ["T3"]}
[root@localhost job] #
3. Data extraction
[root@localhost job] # python2 / root/datax/bin/datax.py job1.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba!
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2019-01-29 14 VMInfo# operatingSystem class 23 15 15 [main] INFO VMInfo-VMInfo# operatingSystem class = > sun.management.OperatingSystemImpl
2019-01-29 14 14 the machine info 23 53. 186 [main] INFO Engine-the machine info = >
OsInfo: Oracle Corporation 1.8 25.151-b12
JvmInfo: Linux amd64 3.10.0-123.el7.x86_64
Cpu num: 8
TotalPhysicalMemory:-0.00G
FreePhysicalMemory:-0.00G
MaxFileDescriptorCount:-1
CurrentOpenFileDescriptorCount:-1
. . .
. . .
2019-01-29 14 2404.012 [job-0] INFO JobContainer-
Mission start time: 2019-01-29 14:23:53
End of mission: 2019-01-29 14:24:04
Total task time: 10s
Average task traffic: 176.48KB/s
Record write speed: 4047rec/s
Total number of read records: 40475
Total number of read and write failures: 0
[root@localhost job] #
4. Validate data
Mysql > select count (1) from T3
+-+
| | count (1) | |
+-+
| | 40475 |
+-+
1 row in set (0.03 sec)
Mysql >
At this point, I believe you have a deeper understanding of "what is the implementation method of Datax offline data extraction". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!
