部署Redis Cluster 6.0 集群并开启密码认证 和 Redis-cluster-proxy负载

部署Redis Cluster集群并开启密码认证

如果只想简单的搭建Redis Cluster，不需要设置密码和公网访问，可以参考官方文档。

节点介绍

Cluster模式推荐最少有6个节点，本次实验搭建了6个节点，使用的端口为7000-7005。Cluster模式是数据分开存放在不同的节点上，如果有6个节点，通常设置3个主节点，每个主节点有一个从节点作为备份。正常情况下读写操作的是主节点，从节点会同步主节点的更变。当某个主节点挂了之后，其对应的从节点会变成主节点。如果一段时间后之前挂掉的主节点恢复了，它将变成从节点。如果某个主节点挂了之后，其对应的从节点也挂了，集群将不可访问。即每个主节点相互独立，从节点作为主节点的备份。

下载和编译

下载地址：https://redis.io/download
下载、解压并编译：

wget http://download.redis.io/releases/redis-6.0.5.tar.gz
tar -zxvf redis-6.0.5.tar.gz
cd cd redis-6.0.5
make

坑：6.x版本的Redis需要新版的gcc才能编译，centOS 7 自带的gcc版本为4.8.5，无法完成编译，需要升级版本。

解决方案：

yum install centos-release-scl
yum install devtoolset-9-gcc*

# 执行如下仅对当前bash生效
scl enable devtoolset-9 bash

# 以上操作仅对当bash生效，若想永久生效，按以下操作：
echo "source /opt/rh/devtoolset-9/enable" >>/etc/profile
source /etc/profile

gcc升级版本之后，需要重新make编译，编译完成后src目录下会生成几个可执行文件redis-benchmark redis-cli redis-sentinel redis-server等。

集群部署

新建6个目录，并将redis.conf和src目录中的redis-server复制到每个目录，目录名称可随意，这里采用redis运行时的端口号作为目录的名称。

cd /data/soft/redis/

mkdir -p /data/soft/redis/conf
cp redis.conf /data/soft/redis/conf/7001_redis.conf
cp redis.conf /data/soft/redis/conf/7002_redis.conf
cp redis.conf /data/soft/redis/conf/7003_redis.conf
cp redis.conf /data/soft/redis/conf/7004_redis.conf
cp redis.conf /data/soft/redis/conf/7005_redis.conf
cp redis.conf /data/soft/redis/conf/7006_redis.conf

cp src/redis-server /data/bin/redis-server
cp src/redis-cli    /data/bin/redis-cli

修改配置

修改conf目录中的*.conf，需要修改配置项如下：

# Redis configuration file.
# ./redis-server /path/to/redis.conf

################################## MODULES #####################################
# include /path/to/local.conf
# loadmodule /path/to/my_module.so

################################## NETWORK #####################################
# By default, if no "bind" configuration directive is specified, Redis listens
# for connections from all the network interfaces available on the server.
bind 192.168.0.239
protected-mode no
port 7005
tcp-backlog 2000
# unixsocket /tmp/redis.sock
# unixsocketperm 700
timeout 0
tcp-keepalive 300

################################# GENERAL #####################################
daemonize yes
supervised no
pidfile /log/redis/redis-7005.pid
# debug, verbose, notice, warning
loglevel notice
logfile /log/redis/redis-7005.log
# syslog-enabled no
# syslog-ident redis
# syslog-facility local0
databases 16
always-show-logo yes

################################ SNAPSHOTTING  ################################
# save <seconds> <changes>
save 600 1
save 300 10
save 250 50
save 100 1000
save 80 5000
save 60 10000

stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump-7005.rdb
dir /log/redis/dump/

################################# REPLICATION #################################
# slaveof 192.168.53.5 7005
masterauth 123456
# slave-serve-stale-data yes
# slave-read-only yes
# repl-diskless-sync no
# repl-ping-slave-period 10
# repl-timeout 60
# repl-disable-tcp-nodelay no
# repl-backlog-size 5mb
# repl-backlog-ttl 3000
# slave-priority 100
# min-slaves-to-write 1
# min-slaves-max-lag 10
# slave-announce-ip 5.5.5.5
# slave-announce-port 1234

################################## SECURITY ###################################
requirepass 123456
# rename-command CONFIG XCONFIG

################################### CLIENTS ####################################
maxclients 10000

############################## MEMORY MANAGEMENT ################################
maxmemory 2048mb
# volatile-lru -> Evict using approximated LRU among the keys with an expire set.
# allkeys-lru -> Evict any key using approximated LRU.
# volatile-lfu -> Evict using approximated LFU among the keys with an expire set.
# allkeys-lfu -> Evict any key using approximated LFU.
# volatile-random -> Remove a random key among the ones with an expire set.
# allkeys-random -> Remove a random key, any key.
# volatile-ttl -> Remove the key with the nearest expire time (minor TTL)
# noeviction -> Don't evict anything, just return an error on write operations.
maxmemory-policy allkeys-lru
maxmemory-samples 5

############################# LAZY FREEING ####################################
# release memory in a non-blocking. DEL, UNLINK and ASYNC
# option of FLUSHALL and FLUSHDB are user-controlled.
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
slave-lazy-flush no

############################## APPEND ONLY MODE ###############################
appendonly yes
appendfilename appendonly-7005.aof
# always|everysec|no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 3mb
aof-load-truncated yes
aof-use-rdb-preamble no

################################ LUA SCRIPTING  ###############################
lua-time-limit 5000

################################ REDIS CLUSTER  ###############################
cluster-enabled yes
cluster-config-file /log/redis/nodes-7005.conf
cluster-node-timeout 15000
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage yes

################################## SLOW LOG ###################################
slowlog-log-slower-than 5000
slowlog-max-len 516

################################ LATENCY MONITOR ##############################
latency-monitor-threshold 0

############################# EVENT NOTIFICATION ##############################
notify-keyspace-events ""

############################### ADVANCED CONFIG ###############################
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
# -5: max size: 64 Kb  <-- not recommended for normal workloads
# -4: max size: 32 Kb  <-- not recommended
# -3: max size: 16 Kb  <-- probably not recommended
# -2: max size: 8 Kb   <-- good
# -1: max size: 4 Kb   <-- good
list-max-ziplist-size -2
# 0|1|2|3
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
# normal -> normal clients including MONITOR clients
# slave  -> slave clients
# pubsub -> clients subscribed to at least one pubsub channel or pattern
# client-output-buffer-limit <class> <hard limit> <soft limit> <soft seconds>
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
# client-query-buffer-limit 1gb
# proto-max-bulk-len 512mb
hz 10
aof-rewrite-incremental-fsync yes
# lfu-log-factor 10
# lfu-decay-time 1

########################### ACTIVE DEFRAGMENTATION #######################
# Enabled active defragmentation
# activedefrag yes

# Minimum amount of fragmentation waste to start active defrag
# active-defrag-ignore-bytes 100mb

# Minimum percentage of fragmentation to start active defrag
# active-defrag-threshold-lower 10

# Maximum percentage of fragmentation at which we use maximum effort
# active-defrag-threshold-upper 100

# Minimal effort for defrag in CPU percentage
# active-defrag-cycle-min 25

# Maximal effort for defrag in CPU percentage
# active-defrag-cycle-max 75

上面的配置文件已经列出，当前为7005的端口，其他端口的配置文件都类似，改下端口既可。

创建集群

启动每个节点：

/data/soft/redis/src/redis-check-aof --fix /log/redis/dump/appendonly-7001.aof

/data/soft/redis/src/redis-check-aof --fix /log/redis/dump/appendonly-7002.aof

/data/soft/redis/src/redis-check-aof --fix /log/redis/dump/appendonly-7003.aof

/data/soft/redis/src/redis-check-aof --fix /log/redis/dump/appendonly-7004.aof

/data/soft/redis/src/redis-check-aof --fix /log/redis/dump/appendonly-7005.aof

/data/soft/redis/src/redis-check-aof --fix /log/redis/dump/appendonly-7006.aof

/data/bin/redis-server /data/soft/redis/conf/redis-7001.conf >/dev/null 2>&1 &

/data/bin/redis-server /data/soft/redis/conf/redis-7002.conf >/dev/null 2>&1 &

/data/bin/redis-server /data/soft/redis/conf/redis-7003.conf >/dev/null 2>&1 &

/data/bin/redis-server /data/soft/redis/conf/redis-7004.conf >/dev/null 2>&1 &

/data/bin/redis-server /data/soft/redis/conf/redis-7005.conf >/dev/null 2>&1 &

/data/bin/redis-server /data/soft/redis/conf/redis-7006.conf >/dev/null 2>&1 &

启动集群：

/data/bin/redis-cli --cluster create ip:7000 ip:7001 ip:7002 ip:7003 ip:7004 ip:7005 --cluster-replicas 1 -a 123456

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.0.239:7006 to 192.168.0.239:7001
Adding replica 192.168.0.239:7003 to 192.168.0.239:7004
Adding replica 192.168.0.239:7005 to 192.168.0.239:7002
M: 4c8fd6640bac7463a517e10fda942568177f1175 192.168.0.239:7001
   slots:[0-5460] (5461 slots) master
M: 8a924d45c5094bca66c736661f267de3ff348134 192.168.0.239:7002
   slots:[10923-16383] (5461 slots) master
S: 9e6a1995558a142002579a1bf2659b0fff6a986b 192.168.0.239:7003
   replicates b8985dcd19735b50a9b13ba8ef2449504b1da755
M: b8985dcd19735b50a9b13ba8ef2449504b1da755 192.168.0.239:7004
   slots:[5461-10922] (5462 slots) master
S: baf6fb8fa06e58e7ca541d4f9c76ee63255e797f 192.168.0.239:7005
   replicates 8a924d45c5094bca66c736661f267de3ff348134
S: 10993fd7b1cdf5d0293373bec821bdb7763426ed 192.168.0.239:7006
   replicates 4c8fd6640bac7463a517e10fda942568177f1175
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...
>>> Performing Cluster Check (using node 192.168.0.239:7001)
M: 4c8fd6640bac7463a517e10fda942568177f1175 192.168.0.239:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 9e6a1995558a142002579a1bf2659b0fff6a986b 192.168.0.239:7003
   slots: (0 slots) slave
   replicates b8985dcd19735b50a9b13ba8ef2449504b1da755
S: 10993fd7b1cdf5d0293373bec821bdb7763426ed 192.168.0.239:7006
   slots: (0 slots) slave
   replicates 4c8fd6640bac7463a517e10fda942568177f1175
M: 8a924d45c5094bca66c736661f267de3ff348134 192.168.0.239:7002
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: baf6fb8fa06e58e7ca541d4f9c76ee63255e797f 192.168.0.239:7005
   slots: (0 slots) slave
   replicates 8a924d45c5094bca66c736661f267de3ff348134
M: b8985dcd19735b50a9b13ba8ef2449504b1da755 192.168.0.239:7004
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

这里的IP可以是公网IP也可以是内网IP，-a后面的参数是设置的访问密码，–cluster-replicas后面的参数是每个主节点对应的从节点的数量。
这时会自动分配主从节点，如果确认分配无误，输入yes即可创建cluster。

坑：在cluster模式中，每个节点除了要使用设置的访问端口以外，还需要使用访问端口加10000的端口号进行数据传输，所以在防火墙开放端口时也需要打开对应的端口，否则会出现wait…的提示。
在本例中，需要开放的端口号为7001-7006和17001-17006。

测试集群：

可以使用redis-cli来连接集群，-c代表使用cluster模式。

/data/soft/redis-cli -c -h ip -p 7000 -a 密码

查看集群信息：

192.168.0.239:7001> cluster info 
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:107
cluster_stats_messages_pong_sent:111
cluster_stats_messages_sent:218
cluster_stats_messages_ping_received:106
cluster_stats_messages_pong_received:107
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:218

测试节点读写信息：

192.168.0.239:7001> set foo bar
-> Redirected to slot [12182] located at 192.168.0.239:7002
OK
192.168.0.239:7002> get foo
"bar"

设置公网访问

现在通过公网访问会出现unknown cluster ip:7000的报错信息，这是因为在nodes.conf文件中某些节点的IP是内网IP。解决方法为如下：

• 停止每个节点的进程。
• 编辑每个节点的nodes.conf文件，将其中的内网IP换成公网IP。
• 重新启动每个节点。

注意：设置公网访问Redis集群属于危险行为，一般生产环境进制使用公网redis集群

配置Redis-cluster-proxy

以下信息来自于官方的说明：

redis-cluster-proxy是Redis集群的代理。Redis能够在基于自动故障转移和分片的集群模式下运行。
这种特殊模式（指Redis集群模式）需要使用特殊的客户端来理解集群协议:通过代理，集群被抽象了出来，可以实现像单实例一样实现redis集群的访问。

Redis集群代理是多线程的，默认情况下，它目前使用多路复用通信模型，这样每个线程都有自己的集群连接，所有属于线程本身的客户端都可以共享该连接。

无论如何，在某些特殊情况下(多事务或阻塞命令)，多路复用被禁用，客户端将拥有自己的集群连接。
通过这种方式，只发送简单命令(比如get和set)的客户端将不需要一组到Redis集群的私有连接。

注：依赖 gcc 4.9 以上版本，所以需要升级gcc版本，前面已经写过了。

下载proxy工具

https://github.com/RedisLabs/redis-cluster-proxy/releases

wget https://github.com/RedisLabs/redis-cluster-proxy/releases/redis-cluster-proxy-1.0-beta2.tar.gz

tar -zxvf redis-cluster-proxy-1.0-beta2.tar.gz

mv redis-cluster-proxy-1.0-beta2 /data/soft/redis-cluster-proxy

cd /data/soft/redis-cluster-proxy/ &&  make install

修改配置文件

修改/data/soft/redis-cluster-proxy目录下proxy.conf文件，该配置文件和redis配置文件相似，具体参数还未整理，后续更新文档

重点修改以下配置：

• 配置监听的集群节点信息

• 可修改代理端口，默认7777

• 可修改线程数，因6.0后是多线程

• 如果后端redis集群配置了密码，需要在auth参数增加后端redis密码

配置文件如下：

# Redis Cluster Proxy configuration file example.
#
# Note that in order to read the configuration file, the proxy must be
# started with the file path passed to the -c option:
#
# ./redis-cluster-proxy -c /path/to/proxy.conf

################################## INCLUDES ###################################

# Include one or more other config files here.  Include files can include
# other files.
#
# If instead you are interested in using includes to override configuration
# options, it is better to use include as the last line.
#
# include /path/to/local.conf
# include /path/to/other.conf

######################## CLUSTER ENTRY POINT ADDRESS ##########################

# Indicate the entry point address in the same way it can be indicated in the
# redis-cluster-proxy command line arguments.
# Note that it can be overridden by the command line argument itself.
# You can also specify multiple entry-points, by adding more lines, ie:
# cluster 127.0.0.1:7000
# cluster 127.0.0.1:7001
# You can also use the "entry-point" alias instead of cluster, ie:
# entry-point 127.0.0.1:7000
#
cluster 192.168.0.239:7001 
cluster 192.168.0.239:7002 
cluster 192.168.0.239:7003 
cluster 192.168.0.239:7004 
cluster 192.168.0.239:7005 
cluster 192.168.0.239:7006

################################### MAIN ######################################

# Set the port used by Redis Cluster Proxy to listen to incoming connections
# from clients (default 7777)
port 7000

# If you want you can bind a single interface, if the bind option is not
# specified all the interfaces will listen for incoming connections.
# You can also bind on multiple interfaces by declaring bind on multiple lines
#
# bind 127.0.0.1

# Specify the path for the Unix socket that will be used to listen for
# incoming connections. There is no default, so Redis Cluster Proxy won't
# listen on a Unix socket when not specified.
#
# unixsocket /path/to/proxy.socket

# Set the Unix socket file permissions (default 0)
#
# unixsocketperm 760

# Set the number of threads.
threads 8

# Set the TCP keep-alive value on the Redis Cluster Proxy's socket
#
# tcpkeepalive 300

# Set the TCP backlog on the Redis Cluster Proxy's socket
#
# tcp-backlog 511


# Size of the connections pool used to provide ready-to-use sockets to
# private connections. The number (size) indicates the number of starting
# connections in the pool.
# Use 0 to disable connections pool at all.
# Every thread will have its pool of ready-to-use connections.
# When the proxy starts, every thread will populate a pool containing
# connections to all the nodes of the cluster.
# Whenever a client needs a private connection, it can take a connection
# from the pool, if available. This will speed-up the client transition from
# the thread's shared connection to its own private connection, since the
# connection from the thread's pool should be already connected and
# ready-to-use. Otherwise, clients with priovate connections must re-connect
# the the nodes of the cluster (this re-connection will act in a 'lazy' way).
#
# connections-pool-size 10

# Minimum number of connections in the the pool. Below this value, the
# thread will start re-spawning connections at the defined rate until
# the pool will be full again.
#
# connections-pool-min-size 10

# Interval in milliseconds used to re-spawn connections in the pool.
# Whenever the number of connections in the pool drops below the minimum
# (see 'connections-pool-min-size' above), the thread will start
# re-spawing connections in the pool, until the pool will be full again.
# New connections will be added at this specified interval.
#
# connections-pool-spawn-every 50

# Number of connections to re-spawn in the pool at every cycle that will
# happen with an interval defined by 'connections-pool-spawn-every' (see above).
#
# connections-pool-spawn-rate 50

# Run Redis Cluster Proxy as a daemon.
daemonize no

# If a pid file is specified, the proxy writes it where specified at startup
# and removes it at exit.
#
# When the proxy runs non daemonized, no pid file is created if none is
# specified in the configuration. When the proxy is daemonized, the pid file
# is used even if not specified, defaulting to
# "/var/run/redis-cluster-proxy.pid".
#
# Creating a pid file is best effort: if the proxy is not able to create it
# nothing bad happens, the server will start and run normally.
#
pidfile /log/redis/redis-cluster-proxy.pid

# Specify the log file name. Also the empty string can be used to force
# Redis Cluster Porxy to log on the standard output. Note that if you use
# standard output for logging but daemonize, logs will be sent to /dev/null
#
logfile "/log/redis/redis-cluster-proxy.log"

# Enable cross-slot queries that can use multiple keys belonging to different
# slots or even different nodes.
# WARN: these queries will break the the atomicity deisgn of many Redis
# commands.
# NOTE: cross-slots queries are not supported by all the commands, even if
# this feature is enabled
#
# enable-cross-slot no

# Maximum number of clients allowed
#
# max-clients 10000

# Authentication password used to authenticate on the cluster in case its nodes
# are password-protected. The password will be used both for fetching cluster's
# configuration and to automatically authenticate proxy's internal connections
# to the cluster itself (both multiplexing shared connections and clients'
# private connections. So, clients connected to the proxy won't need to issue
# the Redis AUTH command in order to be authenticated.
#
auth nfJNhCDwv0hOlyFdbT9XvlkdZq 

# Authentication username (supported by Redis >= 6.0)
#
# auth-user myuser

################################# LOGGING #####################################

# Log level: can be debug, info, success, warning o error.
log-level error

# Dump queries received from clients in the log (log-level debug required)
#
# dump-queries no

# Dump buffer in the log (log-level debug required)
#
# dump-buffer no

# Dump requests' queues (requests to send to cluster, request pending, ...)
# in the log (log-level debug required)
#
# dump-queues no

启动redis-cluster-proxy

第一次先当前控制台启动,可以看下错误输出

/data/bin/redis-cluster-proxy -c /usr/local/redis-cluster-proxy/proxy.conf

第二次用后台启动

/data/bin/redis-cluster-proxy --daemonize -c /usr/local/redis-cluster-proxy/proxy.conf

启动成功后，可以通过ps -ef 查看是否启动

ps -ef|grep redis

然后通过redis客户端连接7000端口（proxy代理端口）

/data/bin/redis-cli -h 127.0.0.1 -p 7000 -a 123456
127.0.0.1:7000> set foo bar1 
OK
127.0.0.1:7000> get foo
"bar1"

redis-cluster-proxy是完美的解决方案？

因为刚推出不久，生产环境基本上不会有太多实际的应用，里面肯定有不少坑，但不妨害对其有更多的期待。
初次尝试可以感受的到，redis-cluster-proxy是一个非常轻量级，清爽简单的proxy代理层，它解决了一些redis cluster存在的一些实际问题，对应于程序来说也带来了一些方便性。

如果没有源码开发能力，相比其他第三方proxy中间件，必须要承认官方可靠性和权威性。

那么，redis-cluster-proxy是一个完美的解决方案么，留下两个问题

1. 如何解决redis-cluster-proxy单点故障？

2. proxy节点的如何面对网络流量风暴？