Redis 集群

• 优点: ⼤⼤降低了扩容时数据搬运的规模, 提⾼了扩容操作的效率.
• 缺点: 数据分配不均匀 (有的多有的少, 数据倾斜)

hash_slot = crc16(key) % 16384 //crc16为一种hash算法

port 6379
bind 0.0.0.0
protected-mode no
appendonly yes
cluster-enabled yes #开启集群
cluster-config-file nodes.conf #集群节点⽣成的配置
cluster-node-timeout 5000 #节点失联的超时时间
cluster-announce-ip 172.30.0.101 #节点⾃⾝ ip
cluster-announce-port 6379 #节点⾃⾝的业务端口，客户端连接的端口
cluster-announce-bus-port 16379 #节点⾃⾝的总线端口，集群管理的信息交互是通过这个端⼝进⾏的.

version: '3.7'
networks:
 mynet:
 ipam:
 config:
 - subnet: 172.30.0.0/24 #创建子网
services:
 redis1:
 image: 'redis:5.0.9'
 container_name: redis1
 restart: always
 volumes:
 - ./redis1/:/etc/redis/
 ports:
 - 6371:6379 #端口映射，方便在容器外通过客户端直接访问
 - 16371:16379
 command:
 redis-server /etc/redis/redis.conf
 networks:
 mynet:
 ipv4_address: 172.30.0.101 #配置静态ip，注意网络号与子网一致
 redis2:
 image: 'redis:5.0.9'
 container_name: redis2
 restart: always
volumes:
 - ./redis2/:/etc/redis/
 ports:
 - 6372:6379
 - 16372:16379
 command:
 redis-server /etc/redis/redis.conf
 networks:
 mynet:
 ipv4_address: 172.30.0.102
 redis3:
 image: 'redis:5.0.9'
 container_name: redis3
 restart: always
 volumes:
 - ./redis3/:/etc/redis/
 ports:
 - 6373:6379
 - 16373:16379
 command:
 redis-server /etc/redis/redis.conf
 networks:
 mynet:
 ipv4_address: 172.30.0.103
 redis4:
 image: 'redis:5.0.9'
 container_name: redis4
 restart: always
 volumes:
 - ./redis4/:/etc/redis/
 ports:
 - 6374:6379
 - 16374:16379
 command:
 redis-server /etc/redis/redis.conf
 networks:
 mynet:
 ipv4_address: 172.30.0.104
 redis5:
 image: 'redis:5.0.9'
 container_name: redis5
 restart: always
 volumes:
 - ./redis5/:/etc/redis/
ports:
 - 6375:6379
 - 16375:16379
 command:
 redis-server /etc/redis/redis.conf
 networks:
 mynet:
 ipv4_address: 172.30.0.105
 redis6:
 image: 'redis:5.0.9'
 container_name: redis6
 restart: always
 volumes:
 - ./redis6/:/etc/redis/
 ports:
 - 6376:6379
 - 16376:16379
 command:
 redis-server /etc/redis/redis.conf
 networks:
 mynet:
 ipv4_address: 172.30.0.106
 redis7:
 image: 'redis:5.0.9'
 container_name: redis7
 restart: always
 volumes:
 - ./redis7/:/etc/redis/
 ports:
 - 6377:6379
 - 16377:16379
 command:
 redis-server /etc/redis/redis.conf
 networks:
 mynet:
 ipv4_address: 172.30.0.107
 redis8:
 image: 'redis:5.0.9'
 container_name: redis8
 restart: always
 volumes:
 - ./redis8/:/etc/redis/
 ports:
 - 6378:6379
- 16378:16379
 command:
 redis-server /etc/redis/redis.conf
 networks:
 mynet:
 ipv4_address: 172.30.0.108
 redis9:
 image: 'redis:5.0.9'
 container_name: redis9
 restart: always
 volumes:
 - ./redis9/:/etc/redis/
 ports:
 - 6379:6379
 - 16379:16379
 command:
 redis-server /etc/redis/redis.conf
 networks:
 mynet:
 ipv4_address: 172.30.0.109
 
 redis10:
 image: 'redis:5.0.9'
 container_name: redis10
 restart: always
 volumes:
 - ./redis10/:/etc/redis/
 ports:
 - 6380:6379
 - 16380:16379
 command:
 redis-server /etc/redis/redis.conf
 networks:
 mynet:
 ipv4_address: 172.30.0.110
 redis11:
 image: 'redis:5.0.9'
 container_name: redis11
 restart: always
 volumes:
 - ./redis11/:/etc/redis/
 ports:
 - 6381:6379
 - 16381:16379
 command:
redis-server /etc/redis/redis.conf
 networks:
 mynet:
 ipv4_address: 172.30.0.111

ps ajx | grep redis

docker-compose up -d

处理流程

• 故障判定

1. 节点 A 给 节点 B 发送 ping 包, B 就会给 A 返回⼀个 pong 包. ping 和 pong 除了 message type 属性之外, 其他部分都是⼀样的. 这⾥包含了集群的配置信息(该节点的id, 该节点从属于哪个分⽚, 是主节点还是从节点, 从属于谁, 持有哪些 slots 的位图...).
2. 每个节点, 每秒钟, 都会给⼀些随机的节点发起 ping 包, ⽽不是全发⼀遍. 这样设定是为了避免在节点很多的时候, ⼼跳包也⾮常多
3. 当节点 A 给节点 B 发起 ping 包, B 不能如期回应的时候, 此时 A 就会尝试重置和 B 的 tcp 连接, 看能否连接成功. 如果仍然连接失败, A 就会把 B 设为 PFAIL 状态(相当于主观下线).
4. A 判定 B 为 PFAIL 之后, 会通过 redis 内置的 Gossip 协议, 和其他节点进⾏沟通, 向其他节点确认 B 的状态. (每个节点都会维护⼀个⾃⼰的 "下线列表", 由于视⻆不同, 每个节点的下线列表也不⼀定相同). 、
5. 此时 A 发现其他很多节点, 也认为 B 为 PFAIL, 并且数⽬超过总集群个数的⼀半, 那么 A 就会把 B 标记成 FAIL (相当于客观下线), 并且把这个消息同步给其他节点(其他节点收到之后, 也会把 B 标记成 FAIL). ⾄此, B 就彻底被判定为故障节点了.

• 故障迁移

1. 从节点判定⾃⼰是否具有参选资格. 如果从节点和主节点已经太久没通信(此时认为从节点的数据和主节点差异太⼤了), 时间超过阈值, 就失去竞选资格.
2. 具有资格的节点, ⽐如 C 和 D, 就会先休眠⼀定时间. 休眠时间 = 500ms 基础时间 + [0, 500ms] 随机时间 + 排名 * 1000ms. offset 的值越⼤, 则排名越靠前(越⼩).
3. ⽐如 C 的休眠时间到了, C 就会给其他所有集群中的节点, 进⾏拉票操作. 但是只有主节点才有投票资格.
4. 主节点就会把⾃⼰的票投给 C (每个主节点只有 1 票). 当 C 收到的票数超过主节点数⽬的⼀半, C 就会晋升成主节点. (C ⾃⼰负责执⾏ slaveof no one, 并且让 D 执⾏ slaveof C).
5. 同时, C 还会把⾃⼰成为主节点的消息, 同步给其他集群的节点. ⼤家也都会更新⾃⼰保存的集群结构信息.

集群扩容

第⼀步: 把新的主节点加入到集群

redis-cli --cluster add-node 172.30.0.110:6379 172.30.0.101:6379

• add-node 后的第⼀组地址是新节点的地址. 第⼆组地址是集群中的任意节点地址.

127.0.0.1:6379> cluster nodes
b3f2ba758318f4bd54031c98c01d7a6155ff43d3 172.30.0.102:6379@16379 master - 0 1683183720184 2 connected 5461-10922
e34911c57d7605903de84ec05b3deac611aaef7e 172.30.0.105:6379@16379 slave 00d319e23ef76a4d51e74600c42ee2a371ae81f6 0 1683183722189 5 connected
522a1bd88a1a9084e6919fa88f4bf1c3655ad837 172.30.0.110:6379@16379 master - 0 1683183720000 10 connected
579282abe81b3f20ffd17d5a1956cdca3b0e71b0 172.30.0.103:6379@16379 master - 0 1683183721084 3 connected 10923-16383
00d319e23ef76a4d51e74600c42ee2a371ae81f6 172.30.0.101:6379@16379 master - 0 1683183721185 1 connected 0-5460
6cf48cc11d0171b6ab1b418808473167acd7986e 172.30.0.106:6379@16379 slave 00d319e23ef76a4d51e74600c42ee2a371ae81f6 0 1683183720000 6 connected
e9ea79b1326ea5a75a1701d5c12a0f6081c1d043 172.30.0.109:6379@16379 myself,slave 579282abe81b3f20ffd17d5a1956cdca3b0e71b0 0 1683183720000 9 connected
fd18c7f164b09ec563f4573ec9d6466e6769221e 172.30.0.108:6379@16379 slave b3f2ba758318f4bd54031c98c01d7a6155ff43d3 0 1683183720684 8 connected
2a248acb47f0036655397897f9800c70ea22514f 172.30.0.104:6379@16379 slave 579282abe81b3f20ffd17d5a1956cdca3b0e71b0 0 1683183722088 4 connected
628d1ec9ceef6760b9038c4fbc83ee92430062ac 172.30.0.107:6379@16379 slave b3f2ba758318f4bd54031c98c01d7a6155ff43d3 0 1683183720000 7 connected

第⼆步: 重新分配 slots

redis-cli --cluster reshard 172.30.0.101:6379

• 多少个 slots 要进⾏ reshard ? (此处我们填写 4096)
• 哪个节点来接收这些 slots ? (此处我们填写 172.30.0.110 这个节点的集群节点 id)
• 这些 slots 从哪些节点搬运过来? (此处我们填写 all, 表⽰从其他所有的节点都进⾏搬运)

How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? 522a1bd88a1a9084e6919fa88f4bf1c3655ad837
Please enter all the source node IDs.
 Type 'all' to use all the nodes as source nodes for the hash slots.
 Type 'done' once you entered all the source nodes IDs.
Source node #1: all

redis-cli --cluster add-node 172.30.0.111:6379 172.30.0.101:6379 --cluster-slave --cluster-master-id [172.30.1.110 节点的 nodeId]