Redis实战案例：基于Redis 4 的 Redis Cluster

准备redis Cluster 基本配置

1. 每个redis 节点采用相同的硬件配置、相同的密码、相同的redis版本
2. 所有redis服务器必须没有任何数据
3. 先启动为单机redis且没有任何key value
4. 准备三台CentOS 7 主机，已编译安装好redis，各启动两个redis实例，分别使用6379和6380端口，从而模拟实现6台redis实例

10.0.0.7:6379|6380 10.0.0.17:6379|6380 10.0.0.27:6379|6380

准备6个实例：在三个主机上重复下面的操作

#做准备6379的实例配置文件 [root@redis-node1 ~]#systemctl stop redis [root@redis-node1 ~]#cd /apps/redis/etc/ [root@redis-node1 etc]#sed -i -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e '/^# masterauth/a masterauth 123456' -e '/# requirepass/a requirepass 123456' -e '/# cluster-enabled yes/a cluster-enabled yes' -e '/# cluster-config-file nodes-6379.conf/a cluster-config-file nodes-6379.conf' -e 's/^dir .*/dir \/apps\/redis\/data/' -e '/appendonly no/c appendonly yes' /apps/redis/etc/redis.conf #准备6380端口的实例的配置文件 [root@redis-node1 etc]#cp -p redis.conf redis6380.conf [root@redis-node1 etc]#sed -i -e 's/6379/6380/' -e 's/dbfilename dump\.rdb/dbfilename dump6380.rdb/' -e 's/appendfilename "appendonly\.aof"/appendfilename "appendonly6380.aof"/' /apps/redis/etc/redis6380.conf #准备服务文件 [root@redis-node1 ~]#cp /lib/systemd/system/redis.service /lib/systemd/system/redis6380.service [root@redis-node1 ~]#sed -i 's/redis.conf/redis6380.conf/' /lib/systemd/system/redis6380.service #启动服务，查看到端口都打开 [root@redis-node1 ~]#systemctl daemon-reload [root@redis-node1 ~]#systemctl start redis redis6380 [root@redis-node1 ~]#ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 128 *:16379 *:* LISTEN 0 128 *:16380 *:* LISTEN 0 128 *:6379 *:* LISTEN 0 128 *:6380 *:* LISTEN 0 128 *:22 *:* LISTEN 0 100 [::1]:25 [::]:* LISTEN 0 128 [::]:22 [::]:* [root@redis-node1 ~]#ps -ef|grep redis redis 71539 1 0 22:13 ? 00:00:00 /apps/redis/bin/redis-server 0.0.0.0:6379 [cluster] redis 71543 1 0 22:13 ? 00:00:00 /apps/redis/bin/redis-server 0.0.0.0:6380 [cluster] root 71553 31781 0 22:15 pts/0 00:00:00 grep --color=auto redis [root@redis-node1 ~]#tree /apps/redis/ /apps/redis/ ├── bin │   ├── redis-benchmark │   ├── redis-check-aof │   ├── redis-check-rdb │   ├── redis-cli │   ├── redis-sentinel -> redis-server │   └── redis-server ├── data │   ├── appendonly6380.aof │   ├── appendonly.aof │   ├── dump6380.rdb │   ├── dump.rdb │   ├── nodes-6379.conf │   └── nodes-6380.conf ├── etc │   ├── redis6380.conf │   ├── redis.conf │   └── redis.conf.bak ├── logs └── run 5 directories, 15 files

准备redis-trib.rb工具

Redis 3和 4版本需要使用到集群管理工具redis-trib.rb，这个工具是redis官方推出的管理redis集群的工具，集成在redis的源码src目录下，是基于redis提供的集群命令封装成简单、便捷、实用的操作工具，redis-trib.rb是redis作者用ruby开发完成的，centos 7 系统yum安装的ruby存在版本较低问题，如下：

[root@redis-node1 ~]#find / -name redis-trib.rb /root/redis-4.0.14/src/redis-trib.rb [root@redis-node1 ~]#cp /root/redis-4.0.14/src/redis-trib.rb /usr/bin/ [root@redis-node1 ~]#redis-trib.rb #缺少ruby环境无法运行rb脚本 /usr/bin/env: ruby: No such file or directory #CentOS 7带的ruby版本过低,无法运行上面ruby脚本，不要安装 [root@redis-node1 ~]#yum install ruby rubygems -y [root@redis-node1 ~]#gem install redis Fetching: redis-4.1.3.gem (100%) ERROR: Error installing redis: redis requires Ruby version >= 2.3.0.

解决ruby版本较低问题：

[root@redis-node1 ~]#yum -y install gcc openssl-devel zlib-devel [root@redis-node1 ~]#wget https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.5.tar.gz [root@redis-node1 ~]#tar xf ruby-2.5.5.tar.gz [root@redis-node1 ~]#cd ruby-2.5.5 [root@redis-node1 ruby-2.5.5]# ./configure [root@redis-node1 ruby-2.5.5]# make -j 2 [root@redis-node1 ruby-2.5.5]# make install [root@redis-node1 ruby-2.5.5]# ruby -v ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux]

redis-trib.rb无法运行错误

[root@redis-node1 ruby-2.5.5]#redis-trib.rb -h Traceback (most recent call last): 2: from /usr/bin/redis-trib.rb:25:in `<main>' 1: from /usr/local/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require' /usr/local/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- redis (LoadError)

解决上述错误：

[root@redis-node1 ruby-2.5.5]#gem install redis Fetching: redis-4.1.3.gem (100%) Successfully installed redis-4.1.3 Parsing documentation for redis-4.1.3 Installing ri documentation for redis-4.1.3 Done installing documentation for redis after 1 seconds 1 gem installed

如果无法在线安装，可以下载reids模块安装包离线安装

#https://rubygems.org/gems/redis #先下载redis模块安装包 [root@redis-node1 ~]#gem install -l redis-3.3.0.gem #安装redis模块

redis-trib.rb命令用法

[root@redis-node1 ~]#redis-trib.rb Usage: redis-trib <command> <options> <arguments ...> create host1:port1 ... hostN:portN #创建集群 --replicas <arg> #指定master的副本数量 check host:port #检查集群信息 info host:port #查看集群主机信息 fix host:port #修复集群 --timeout <arg> reshard host:port #在线热迁移集群指定主机的slots数据 --from <arg> --to <arg> --slots <arg> --yes --timeout <arg> --pipeline <arg> rebalance host:port #平衡集群中各主机的slot数量 --weight <arg> --auto-weights --use-empty-masters --timeout <arg> --simulate --pipeline <arg> --threshold <arg> add-node new_host:new_port existing_host:existing_port #添加主机到集群 --slave --master-id <arg> del-node host:port node_id #删除主机 set-timeout host:port milliseconds #设置节点的超时时间 call host:port command arg arg .. arg #在集群上的所有节点上执行命令 import host:port #导入外部redis服务器的数据到当前集群 --from <arg> --copy --replace help (show this help)

修改密码redis 登录密码

#修改redis-trib.rb连接redis的密码 [root@redis ~]#vim /usr/local/lib/ruby/gems/2.5.0/gems/redis-4.1.3/lib/redis/client.rb

创建redis cluster集群

#确保三台主机6个实例都启动状态 [root@redis-node1 ~]#systemctl is-active redis redis6380 active active [root@redis-node2 ~]#systemctl is-active redis redis6380 active active [root@redis-node3 ~]#systemctl is-active redis redis6380 active active #--replicas 1 表示一个slave 节点 [root@redis-node1 ~]#redis-trib.rb create --replicas 1 10.0.0.7:6379 10.0.0.7:6380 10.0.0.17:6379 10.0.0.17:6380 10.0.0.27:6379 10.0.0.27:6380 >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 10.0.0.7:6379 10.0.0.17:6379 10.0.0.27:6379 Adding replica 10.0.0.17:6380 to 10.0.0.7:6379 Adding replica 10.0.0.27:6380 to 10.0.0.17:6379 Adding replica 10.0.0.7:6380 to 10.0.0.27:6379 M: 739cb4c9895592131de418b8bc65990f81b75f3a 10.0.0.7:6379 slots:0-5460 (5461 slots) master S: 0e0beba04cc98da02ebdb5225a11b84aa8062e10 10.0.0.7:6380 replicates a01fd3d81922d6752f7c960f1a75b6e8f28d911b M: dddabb4e19235ec02ae96ab2ce67e295ce0274d7 10.0.0.17:6379 slots:5461-10922 (5462 slots) master S: 34708909088ba562decbc1525a9606e088bdddf1 10.0.0.17:6380 replicates 739cb4c9895592131de418b8bc65990f81b75f3a M: a01fd3d81922d6752f7c960f1a75b6e8f28d911b 10.0.0.27:6379 slots:10923-16383 (5461 slots) master S: aefc6203958859024b8383b2fdb87b9e09411ccd 10.0.0.27:6380 replicates dddabb4e19235ec02ae96ab2ce67e295ce0274d7 Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join.... >>> Performing Cluster Check (using node 10.0.0.7:6379) M: 739cb4c9895592131de418b8bc65990f81b75f3a 10.0.0.7:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: 0e0beba04cc98da02ebdb5225a11b84aa8062e10 10.0.0.7:6380 slots: (0 slots) slave replicates a01fd3d81922d6752f7c960f1a75b6e8f28d911b S: 34708909088ba562decbc1525a9606e088bdddf1 10.0.0.17:6380 slots: (0 slots) slave replicates 739cb4c9895592131de418b8bc65990f81b75f3a S: aefc6203958859024b8383b2fdb87b9e09411ccd 10.0.0.27:6380 slots: (0 slots) slave replicates dddabb4e19235ec02ae96ab2ce67e295ce0274d7 M: a01fd3d81922d6752f7c960f1a75b6e8f28d911b 10.0.0.27:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) M: dddabb4e19235ec02ae96ab2ce67e295ce0274d7 10.0.0.17:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.

如果有之前的操作导致Redis集群创 建报错，则执行清空数据和集群命令：

127.0.0.1:6379> FLUSHALL OK 127.0.0.1:6379> cluster reset OK

创建redis cluster集群状态

自动生成配置文件记录master/slave对应关系

[root@redis-node1 ~]#cat /apps/redis/data/nodes-6379.conf 0e0beba04cc98da02ebdb5225a11b84aa8062e10 10.0.0.7:6380@16380 slave a01fd3d81922d6752f7c960f1a75b6e8f28d911b 0 1582383256000 5 connected 34708909088ba562decbc1525a9606e088bdddf1 10.0.0.17:6380@16380 slave 739cb4c9895592131de418b8bc65990f81b75f3a 0 1582383256216 4 connected aefc6203958859024b8383b2fdb87b9e09411ccd 10.0.0.27:6380@16380 slave dddabb4e19235ec02ae96ab2ce67e295ce0274d7 0 1582383257000 6 connected 739cb4c9895592131de418b8bc65990f81b75f3a 10.0.0.7:6379@16379 myself,master - 0 1582383256000 1 connected 0-5460 a01fd3d81922d6752f7c960f1a75b6e8f28d911b 10.0.0.27:6379@16379 master - 0 1582383258230 5 connected 10923-16383 dddabb4e19235ec02ae96ab2ce67e295ce0274d7 10.0.0.17:6379@16379 master - 0 1582383257223 3 connected 5461-10922 vars currentEpoch 6 lastVoteEpoch 0 [root@redis-node1 ~]#

查看状态

[root@redis-node1 ~]#redis-trib.rb info 10.0.0.7:6379 10.0.0.7:6379 (739cb4c9...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.27:6379 (a01fd3d8...) -> 0 keys | 5461 slots | 1 slaves. 10.0.0.17:6379 (dddabb4e...) -> 0 keys | 5462 slots | 1 slaves. [OK] 0 keys in 3 masters. 0.00 keys per slot on average. [root@redis-node1 ~]#redis-trib.rb check 10.0.0.7:6379 >>> Performing Cluster Check (using node 10.0.0.7:6379) M: 739cb4c9895592131de418b8bc65990f81b75f3a 10.0.0.7:6379 slots:0-5460 (5461 slots) master 1 additional replica(s) S: 0e0beba04cc98da02ebdb5225a11b84aa8062e10 10.0.0.7:6380 slots: (0 slots) slave replicates a01fd3d81922d6752f7c960f1a75b6e8f28d911b S: 34708909088ba562decbc1525a9606e088bdddf1 10.0.0.17:6380 slots: (0 slots) slave replicates 739cb4c9895592131de418b8bc65990f81b75f3a S: aefc6203958859024b8383b2fdb87b9e09411ccd 10.0.0.27:6380 slots: (0 slots) slave replicates dddabb4e19235ec02ae96ab2ce67e295ce0274d7 M: a01fd3d81922d6752f7c960f1a75b6e8f28d911b 10.0.0.27:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) M: dddabb4e19235ec02ae96ab2ce67e295ce0274d7 10.0.0.17:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@redis-node1 ~]#redis-cli -a 123456 Warning: Using a password with '-a' option on the command line interface may not be safe. 127.0.0.1:6379> CLUSTER INFO cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3 cluster_current_epoch:6 cluster_my_epoch:1 cluster_stats_messages_ping_sent:252 cluster_stats_messages_pong_sent:277 cluster_stats_messages_sent:529 cluster_stats_messages_ping_received:272 cluster_stats_messages_pong_received:252 cluster_stats_messages_meet_received:5 cluster_stats_messages_received:529 127.0.0.1:6379> [root@redis-node1 ~]#redis-cli -a 123456 -p 6379 INFO replication Warning: Using a password with '-a' option on the command line interface may not be safe. # Replication role:master connected_slaves:1 slave0:ip=10.0.0.17,port=6380,state=online,offset=196,lag=0 master_replid:4ee36f9374c796ca4c65a0f0cb2c39304bb2e9c9 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:196 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:196 [root@redis-node1 ~]#redis-cli -a 123456 -p 6380 INFO replication Warning: Using a password with '-a' option on the command line interface may not be safe. # Replication role:slave master_host:10.0.0.27 master_port:6379 master_link_status:up master_last_io_seconds_ago:2 master_sync_in_progress:0 slave_repl_offset:224 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:dba41cb31c14de7569e597a3d8debc1f0f114c1e master_replid2:0000000000000000000000000000000000000000 master_repl_offset:224 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:224

python脚本实现RedisCluster集群写入

[root@redis-node1 ~]#yum -y install python3 [root@redis-node1 ~]#pip3 install redis-py-cluster [root@redis-node1 ~]#vim redis_cluster_test.py [root@redis-node1 ~]#cat ./redis_cluster_test.py #!/usr/bin/env python3 from rediscluster import RedisCluster startup_nodes = [ {"host":"10.0.0.7", "port":6379}, {"host":"10.0.0.7", "port":6380}, {"host":"10.0.0.17", "port":6379}, {"host":"10.0.0.17", "port":6380}, {"host":"10.0.0.27", "port":6379}, {"host":"10.0.0.27", "port":6380} ] redis_conn= RedisCluster(startup_nodes=startup_nodes,password='123456', decode_responses=True) for i in range(0, 10000): redis_conn.set('key'+str(i),'value'+str(i)) print('key'+str(i)+':',redis_conn.get('key'+str(i))) [root@redis-node1 ~]#chmod +x redis_cluster_test.py [root@redis-node1 ~]#./redis_cluster_test.py ...... key9998: value9998 key9999: value9999

验证脚本写入的状态

[root@redis-node1 ~]#redis-cli -a 123456 -h 10.0.0.7 DBSIZE Warning: Using a password with '-a' option on the command line interface may not be safe. (integer) 3331 [root@redis-node1 ~]#redis-cli -a 123456 -h 10.0.0.17 DBSIZE Warning: Using a password with '-a' option on the command line interface may not be safe. (integer) 3340 [root@redis-node1 ~]#redis-cli -a 123456 -h 10.0.0.27 DBSIZE Warning: Using a password with '-a' option on the command line interface may not be safe. (integer) 3329 [root@redis-node1 ~]#redis-cli -a 123456 GET key1 Warning: Using a password with '-a' option on the command line interface may not be safe. (error) MOVED 9189 10.0.0.17:6379 [root@redis-node1 ~]#redis-cli -a 123456 GET key2 Warning: Using a password with '-a' option on the command line interface may not be safe. "value2" [root@redis-node1 ~]#redis-cli -a 123456 -h 10.0.0.17 GET key1 Warning: Using a password with '-a' option on the command line interface may not be safe. "value1" [root@redis-node1 ~]#redis-trib.rb info 10.0.0.7:6379 10.0.0.7:6379 (739cb4c9...) -> 3331 keys | 5461 slots | 1 slaves. 10.0.0.27:6379 (a01fd3d8...) -> 3329 keys | 5461 slots | 1 slaves. 10.0.0.17:6379 (dddabb4e...) -> 3340 keys | 5462 slots | 1 slaves. [OK] 10000 keys in 3 masters. 0.61 keys per slot on average. [root@redis-node1 ~]#

模拟master故障，对应的slave节点自动提升为新master

[root@redis-node1 ~]#systemctl stop redis [root@redis-node1 ~]#redis-trib.rb check 10.0.0.27:6379 >>> Performing Cluster Check (using node 10.0.0.27:6379) M: a01fd3d81922d6752f7c960f1a75b6e8f28d911b 10.0.0.27:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: aefc6203958859024b8383b2fdb87b9e09411ccd 10.0.0.27:6380 slots: (0 slots) slave replicates dddabb4e19235ec02ae96ab2ce67e295ce0274d7 S: 0e0beba04cc98da02ebdb5225a11b84aa8062e10 10.0.0.7:6380 slots: (0 slots) slave replicates a01fd3d81922d6752f7c960f1a75b6e8f28d911b M: 34708909088ba562decbc1525a9606e088bdddf1 10.0.0.17:6380 slots:0-5460 (5461 slots) master 0 additional replica(s) M: dddabb4e19235ec02ae96ab2ce67e295ce0274d7 10.0.0.17:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. [root@redis-node1 ~]#tail /var/log/messages Feb 22 23:23:13 centos7 redis-server: 71887:M 22 Feb 23:23:13.656 * Saving the final RDB snapshot before exiting. Feb 22 23:23:13 centos7 systemd: Stopped Redis persistent key-value database. Feb 22 23:23:13 centos7 redis-server: 71887:M 22 Feb 23:23:13.660 * DB saved on disk Feb 22 23:23:13 centos7 redis-server: 71887:M 22 Feb 23:23:13.660 * Removing the pid file. Feb 22 23:23:13 centos7 redis-server: 71887:M 22 Feb 23:23:13.660 # Redis is now ready to exit, bye bye... Feb 22 23:23:13 centos7 systemd: Unit redis.service entered failed state. Feb 22 23:23:13 centos7 systemd: redis.service failed. Feb 22 23:23:30 centos7 redis-server: 72046:S 22 Feb 23:23:30.077 * FAIL message received from dddabb4e19235ec02ae96ab2ce67e295ce0274d7 about 739cb4c9895592131de418b8bc65990f81b75f3a Feb 22 23:23:30 centos7 redis-server: 72046:S 22 Feb 23:23:30.077 # Cluster state changed: fail Feb 22 23:23:30 centos7 redis-server: 72046:S 22 Feb 23:23:30.701 # Cluster state changed: ok [root@redis-node1 ~]#redis-trib.rb info 10.0.0.27:6379 10.0.0.27:6379 (a01fd3d8...) -> 3329 keys | 5461 slots | 1 slaves. 10.0.0.17:6380 (34708909...) -> 3331 keys | 5461 slots | 0 slaves. 10.0.0.17:6379 (dddabb4e...) -> 3340 keys | 5462 slots | 1 slaves. [OK] 10000 keys in 3 masters. 0.61 keys per slot on average.

将故障的master恢复后，该节点自动加入集群成为新的slave

[root@redis-node1 ~]#systemctl start redis [root@redis-node1 ~]#redis-trib.rb check 10.0.0.27:6379 >>> Performing Cluster Check (using node 10.0.0.27:6379) M: a01fd3d81922d6752f7c960f1a75b6e8f28d911b 10.0.0.27:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: aefc6203958859024b8383b2fdb87b9e09411ccd 10.0.0.27:6380 slots: (0 slots) slave replicates dddabb4e19235ec02ae96ab2ce67e295ce0274d7 S: 739cb4c9895592131de418b8bc65990f81b75f3a 10.0.0.7:6379 slots: (0 slots) slave replicates 34708909088ba562decbc1525a9606e088bdddf1 S: 0e0beba04cc98da02ebdb5225a11b84aa8062e10 10.0.0.7:6380 slots: (0 slots) slave replicates a01fd3d81922d6752f7c960f1a75b6e8f28d911b M: 34708909088ba562decbc1525a9606e088bdddf1 10.0.0.17:6380 slots:0-5460 (5461 slots) master 1 additional replica(s) M: dddabb4e19235ec02ae96ab2ce67e295ce0274d7 10.0.0.17:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.