Redis集群数据迁移

作者: 轻飘飘D | 来源:发表于2021-03-10 00:35 被阅读0次

redis技术栈
Redis 异构集群数据迁移
redis集群数据迁移
Redis集群数据迁移
hdfs文件迁移
当Redis数据磁盘坏掉之后会发生什么
Redis Cluster数据迁移单实例方案实战
redis迁移工具参考说明
一篇文章教你自建hadoop集群迁移到EMR
Redis集群间数据迁移与对比

1.下载RedisShake

[root@mpb201 ~]# cd /usr/local/src

[root@mpb201 src]# wget https://github.com/alibaba/RedisShake/releases/download/release-v2.0.3-20200724/redis-shake-v2.0.3.tar.gz

[root@mpb201 src]# tar -xvf redis-shake-v2.0.3.tar.gz

[root@mpb201 src]# cd redis-shake-v2.0.3

[root@mpb201 redis-shake-v2.0.3]# ls
ChangeLog  redis-shake.conf  redis-shake.darwin  redis-shake.linux  redis-shake.windows  start.sh  stop.sh

2.获取源集群的Master节点和IP
redis-cli -h {redis_address} -p {redis_port} cluster nodes

[root@MPPAY-CORE2 ~]# redis-cli -h 10.0.101.5 -c -p 7000 cluster nodes
e1dff2cf251fb181d17e3f1198b631167010284b 10.0.101.213:7000 master - 0 1615171002171 295 connected 4096-8191
7ef1467f755f3a0be3c42397966589ebf5b80d62 10.0.101.5:7003 master - 0 1615170999661 13 connected 8192-12287
e4347dda7a22f004658a2591d2ca4b1dd1006010 10.0.101.5:7001 master - 0 1615170998157 294 connected 12288-16383
e3ca082dfdba88c793615d6538c64c3e68b86fe0 10.0.101.5:7000 myself,master - 0 0 0 connected 0-4095
...

3.获取目标集群的Master节点和IP

[root@mpb201 redis-shake-v2.0.3]#  redis-cli -h 10.0.102.201 -c -p 7000 cluster nodes
fca1f250b9fb15a5166470e438e21ef45b1c878e 10.0.102.204:7000@17000 master - 0 1615171549000 4 connected 12288-16383
f1a4f1f596dd923dd02289c3a2a94692d0a18c56 10.0.102.201:7000@17000 myself,master - 0 1615171547000 1 connected 0-4095
fe1ad8daace996a8563d7efa0f84ab0f6f0e2efd 10.0.102.203:7000@17000 master - 0 1615171549000 3 connected 8192-12287
2905b1a509b3ba4367b1bee857da2995c189d750 10.0.102.202:7000@17000 master - 0 1615171550000 2 connected 4096-8191
...

4.编辑RedisShake配置文件

[root@mpb201 redis-shake-v2.0.3]# mkdir log

[root@mpb201 redis-shake-v2.0.3]# pwd
/usr/local/src/redis-shake-v2.0.3

vim redis-shake.conf
------------------------------------------------------------------------------
log.file = /usr/local/src/redis-shake-v2.0.3/log/redis-shake.log
parallel = 64
source.type = cluster
source.address = 10.0.101.5:7000;10.0.101.5:7001;10.0.101.5:7003;10.0.101.213:7000    #源端集群master节点
source.password_raw = #有密码则设置，无则置空
target.type = cluster
target.address = 10.0.102.201:7000;10.0.102.202:7000;10.0.102.203:7000;10.0.102.204:7000    #目标集群master节点
target.password_raw = #有密码则设置，无则置空
target.rdb.output = local_dump
key_exists = rewrite

5.在线迁移

[root@mpb201 redis-shake-v2.0.3]# nohup ./redis-shake.linux -type sync -conf redis-shake.conf &

tail -1000f nohup.out                                              --查看程序启动有没有报错

tail -1000f /usr/local/src/redis-shake-v2.0.3/log/redis-shake.log  --查看redis同步数据信息
出现如下信息，代表全量数据同步完成，进入增量同步阶段  "sync rdb done."
------------------------------------------------------------
2021/03/08 11:58:55 [INFO] DbSyncer[3] total = 37.94MB -      34.95MB [ 92%]  entry=358550      
2021/03/08 11:58:55 [INFO] DbSyncer[3] total = 37.94MB -      37.94MB [100%]  entry=389043      
2021/03/08 11:58:55 [INFO] DbSyncer[3] sync rdb done
2021/03/08 11:58:55 [INFO] DbSyncer[3] FlushEvent:IncrSyncStart Id:redis-shake  
2021/03/08 11:58:55 [INFO] input password is empty, skip auth address[10.0.101.213:7000] with type[auth].
2021/03/08 11:58:56 [INFO] DbSyncer[2] total = 38.34MB -      36.64MB [ 95%]  entry=371800      
2021/03/08 11:58:56 [INFO] DbSyncer[1] total = 37.88MB -      34.69MB [ 91%]  entry=356008      
2021/03/08 11:58:56 [INFO] DbSyncer[0] total = 38.29MB -      37.59MB [ 98%]  entry=378514      
2021/03/08 11:58:56 [INFO] DbSyncer[0] total = 38.29MB -      38.29MB [100%]  entry=386756      
2021/03/08 11:58:56 [INFO] DbSyncer[0] sync rdb done
2021/03/08 11:58:56 [INFO] DbSyncer[0] FlushEvent:IncrSyncStart Id:redis-shake  
2021/03/08 11:58:56 [INFO] input password is empty, skip auth address[10.0.101.5:7000] with type[auth].
2021/03/08 11:58:56 [INFO] DbSyncer[2] total = 38.34MB -      38.34MB [100%]  entry=390403      
2021/03/08 11:58:56 [INFO] DbSyncer[2] sync rdb done
2021/03/08 11:58:56 [INFO] DbSyncer[2] FlushEvent:IncrSyncStart Id:redis-shake  
2021/03/08 11:58:56 [INFO] input password is empty, skip auth address[10.0.101.5:7003] with type[auth].
2021/03/08 11:58:56 [INFO] DbSyncer[1] total = 37.88MB -      37.88MB [100%]  entry=389915      
2021/03/08 11:58:56 [INFO] DbSyncer[1] sync rdb done
2021/03/08 11:58:56 [INFO] DbSyncer[1] FlushEvent:IncrSyncStart Id:redis-shake  
2021/03/08 11:58:56 [INFO] input password is empty, skip auth address[10.0.101.5:7001] with type[auth].
2021/03/08 11:58:56 [INFO] DbSyncer[3] sync:  +forwardCommands=9      +filterCommands=0      +writeBytes=513
2021/03/08 11:58:57 [INFO] DbSyncer[0] sync:  +forwardCommands=5      +filterCommands=0      +writeBytes=179
2021/03/08 11:58:57 [INFO] DbSyncer[2] sync:  +forwardCommands=4      +filterCommands=0      +writeBytes=78
2021/03/08 11:58:57 [INFO] DbSyncer[1] sync:  +forwardCommands=4      +filterCommands=0      +writeBytes=465

...

2021/03/08 12:06:01 [INFO] DbSyncer[3] sync:  +forwardCommands=4      +filterCommands=0      +writeBytes=649
2021/03/08 12:06:01 [INFO] DbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
2021/03/08 12:06:01 [INFO] DbSyncer[2] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
2021/03/08 12:06:02 [INFO] DbSyncer[1] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
2021/03/08 12:06:02 [INFO] DbSyncer[3] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
2021/03/08 12:06:02 [INFO] DbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0

------------------------------------------------------------
执行日志出现如下信息时，代表增量同步无新增内容：
sync:  +forwardCommands=0  +filterCommands=0  +writeBytes=0

6.如果确定已经全部同步完成，且后续不会有新的业务数据进来，则手动停止同步程序

[root@mpb201 redis-shake-v2.0.3]# ps -ef|grep redis-shake.linux
root      3437  3334 11 11:58 pts/0    00:01:04 ./redis-shake.linux -type sync -conf redis-shake.conf


kill -9 3437

7.驗證

[root@MPPAY-CORE2 ~]# 
redis-cli -h 10.0.101.5 -p 7000  info keyspace
redis-cli -h 10.0.101.5 -p 7001  info keyspace
redis-cli -h 10.0.101.5 -p 7003  info keyspace
redis-cli -h 10.0.101.213 -p 7000  info keyspace

redis-cli -h 10.0.102.201 -p 7000  info keyspace
redis-cli -h 10.0.102.202 -p 7000  info keyspace
redis-cli -h 10.0.102.203 -p 7000  info keyspace
redis-cli -h 10.0.102.204 -p 7000  info keyspace