MySQL轻量数据同步工具比较

Posted on Mon 26 November 2012 in 我用(IT)

Background

Context

新业务部署在AWS上,把MySQL放到RDS中了,从性能测试结果上看还是很满意的。业务自身需要实时与主IDC(国内机房)的数据库保持同步(涉及到几十张表,百兆以上的数据量,实时性要求也很高,幸亏只要求单向),原来的技术方案采用系统级的Master-Slaver复制,于是老革命遇到了新问题: 由于RDS目前并不开放系统级的API,使得利用系统IO(例如DRBD的HA方案)或基于binary log(master-slaver)进行复制的系统级方案无法应用。 要么找基于MySQL自有接口的轻量级工具完成同步,要么改应用逻辑自己实现。 后者是个工作量不小的活计,所以我希望能找一个现成儿的工具。

http://aws.amazon.com/rds/faqs/#103

Q: Can I directly access the binary logs for my Database Instance to manage my own replication? Amazon RDS does not currently provide access to the binary logs for your Database Instance.

http://stackoverflow.com/questions/11553371/local-replica-of-rds-database

you can create replicas of an RDS but only as another RDS. You can't do a replica on an EC2 or a local machine.

More info can be found at AWS Management Console or Amazon RDS APIs

More Resources

Deploying a MySQL Database Using EC2

Amazon RDS Resources

MySQL Replication on RDS: 1. Multi-AZ: high available 2. Read Replicas: raizsing ability for reading

MySQL Data Synchronization Tools

SQLyog Job Agent

http://code.google.com/p/sqlyog/downloads/list

这个工具只支持对有主键或唯一键设置的表进行同步,否则报错如下 mysql> desc imsi; +---------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------+--------------+------+-----+---------+-------+ | imsi | varchar(20) | YES | | NULL | | | country | varchar(100) | YES | | NULL | | +---------+--------------+------+-----+---------+-------+ 2 rows in set (0.00 sec)

WARNING: table imsi does not have a Primary Key or a Unique Key with columns defined NOT NULL. Data Sync will operate in a special mode. Refer to documentation for details. Check sja.log for complete error details. 0 0 0

Total time taken - 1 sec(s)

同步两张带主键的表: ./sja job.xml SQLyog Job Agent Version 10.3 Copyright(c) Webyog Inc. All Rights Reserved.

Sync started at Mon Nov 26 12:05:54 2012

Table SrcRows TgtRows Inserted Updated Deleted ========================= ======= ======= ======== ======= ======= tb_sns_setting 41 8 33 2 0 tb_sns_list 123 123 0 0 0

Total time taken - 0 sec(s)

a sample of job.xml

<?xml version="1.0" encoding="UTF-8"?>

    
        
        
        
        
        
        
            172.27..137.253
            app
            tnbl
            globalsetting
        
        
        
            localhost
            root
            
            3306
            globalsetting
        
        
        
            sns_blog
            
tb_sns_list

同步打表测试(tb_global_user,28万条记录): 每次耗时4分钟左右

Percona Toolkit

http://www.percona.com/software/percona-toolkit

同步大数据量测试:

第一次同步:替换了28万多条记录,用时12分钟 $ ./pt-table-sync --execute --verbose --no-check-slave --databases globaldata --tables tb_global_user h=172.27.137.253,u=app,p=tnbl,P=3306
h=172.27.233.218,u=pig,p=,P=3306

Syncing P=3306,h=172.27.233.218,p=...,u=pig

DELETE REPLACE INSERT UPDATE ALGORITHM START END EXIT DATABASE.TABLE

0 289106 0 13282 Chunk 16:44:54 16:58:21 2 globaldata.tb_global_user

第二次同步,更新6千条记录,用时54秒 $ ./pt-table-sync --execute --verbose --no-check-slave --databases globaldata --tables tb_global_user h=172.27.137.253,u=app,p=tnbl,P=3306 h=172.27.233.218,u=pig,p=,P=3306

Syncing P=3306,h=172.27.233.218,p=...,u=pig

DELETE REPLACE INSERT UPDATE ALGORITHM START END EXIT DATABASE.TABLE

0 0 0 6163 Chunk 17:08:11 17:09:05 2 globaldata.tb_global_user

第三次同步,用时36秒 $ ./pt-table-sync --execute --verbose --no-check-slave --databases globaldata --tables tb_global_user h=172.27.137.253,u=app,p=tnbl,P=3306 h=172.27.233.218,u=pig,p=,P=3306

Syncing P=3306,h=172.27.233.218,p=...,u=pig

DELETE REPLACE INSERT UPDATE ALGORITHM START END EXIT DATABASE.TABLE

0 0 0 6163 Chunk 17:15:49 17:16:25 2 globaldata.tb_global_user