The method and steps of synchronizing MySQL to Redis with Canal and RocketMQ

This article mainly explains the method and steps of synchronizing MySQL to Redis with Canal and RocketMQ. Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn the steps of synchronizing MySQL to Redis with Canal and RocketMQ.

First, install Canal

Docker installs stand-alone Canal

Second, install RocketMQ

Docker installs stand-alone RocketMQ

3. Configure MQ of Canal

Official document

3.1 modify Canal Server configuration

# common argument # # canal admin configcanal.admin.manager = 192.168.25.142:8089canal.admin.port = 11110canal.admin.user = admincanal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441canal.serverMode = RocketMQ### MQ # canal.mq.servers = 192.168.25.142:9876canal.mq.retries = 0canal.mq.batchSize = 16384canal.mq.maxRequestSize = 1048576canal.mq.lingerMs = 100canal.mq.bufferMemory = 33554432canal.mq.canalBatchSize = 50canal.mq.canalGetTimeout = 100canal.mq.flatMessage = truecanal.mq.compressionType = nonecanal.mq.acks = all#canal.mq.properties. = canal.mq.producerGroup = test# Set this value to "cloud" If you want open message trace feature in aliyun.canal.mq.accessChannel = local# aliyun mq namespace#canal.mq.namespace = # Kafka Kerberos Info # # # canal.mq.kafka.kerberos.enable = falsecanal.mq.kafka.kerberos.krb5FilePath = ".. / conf/kerberos/krb5.conf" canal.mq.kafka.kerberos.jaasFilePath = ".. / conf/kerberos/jaas.conf"

After the modification, click Save.

3.2Check the instance configuration

Intance.properties does not need to be modified. The configuration is listed here to make it convenient for everyone to avoid the pit.

# mysql serverId V1.0.26 + will autoGen# canal.instance.mysql.slaveId=0# enable gtid use true/falsecanal.instance.gtidon=false# position infocanal.instance.master.address=192.168.25.1:3306canal.instance.master.journal.name=canal.instance.master.position=canal.instance.master.timestamp=canal.instance.master.gtid=# rds oss binlogcanal.instance.rds.accesskey=canal.instance.rds.secretkey=canal.instance.rds.instanceId=# table meta tsdb infocanal.instance.tsdb.enable=true#canal.instance.tsdb.url=jdbc:mysql://127.0.0 .1: 3306/canal_tsdb#canal.instance.tsdb.dbUsername=canal#canal.instance.tsdb.dbPassword=canal#canal.instance.standby.address = # canal.instance.standby.journal.name = # canal.instance.standby.position = # canal.instance.standby.timestamp = # canal.instance.standby.gtid=# username/passwordcanal.instance.dbUsername=canalcanal.instance.dbPassword=canalcanal.instance.connectionCharset = UTF-8# enable druid Decrypt database passwordcanal.instance.enableDruid=false#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==# table regexcanal.instance.filter.regex=. *\.. # table black regexcanal.instance.filter.black.regex=# table field filter (format: schema1.tableName1:field1/field2 Schema2.tableName2:field1/field2) # canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch# table field black filter (format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2) # canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch# mq configcanal.mq.topic=example# dynamic topic route by schema or table regexcanal.mq.dynamicTopic=test.user,student\.. * . *\.. * canal.mq.partition=0# hash partition configcanal.mq.partitionsNum=3canal.mq.partitionHash=test.users:uid,.*\\.. * # IV. Restart Canal Server and Canal Instance

View Canal Server Log

View Canal Instance Log

Start the Canal test RocketExample5.1 pull replacement code $git clone https://github.com/alibaba/canal.git5.2 start the test code

Modify RocketMQ configuration

Start the Main method and view the console

Note: running now... Does not mean that the connection to Broker is successful, it has been stuck here for a long time.

6. Test and send 6.1 MySQL insert data

6.2 View the RocketMQ Operations console

Click Message Detail

6.3 View the RocketMQ Consume console

At this point, I believe you have a deeper understanding of the "methods and steps of synchronizing MySQL to Redis with Canal combined with RocketMQ". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!