弄清楚了多线程的本质之后,就会有一系列的问题,多线程会比单线程有多大的提升?设置多少个线程合适?见一些大神测试过(目前全网也只有美图做过unstable版本的测试,所有的转载都是来源于这个测试),其结果也非常理想,但只是看看也不太过瘾,决定一试为快,本文将对Redis的多线程进行一个粗浅的测试验证。同时需要思考另外一个问题:面对多线程版本的Redis,和Redis cluster,该如何选择?
多线程Redis
redis 6.0 的“多线程”特性让很多标题党高潮连连,参考图片源自于:美图技术团队侵删,核心的线程(Execute Command)还是单线程,多线程是指网络IO(socket)读写的多线程化。
如下图所示,读写网络socket中的数据是可以用多个线程,所以Redis的多线程也叫做io thread,相关参数:“io-threads”。另一个参数是io-threads-do-reads,这里涉及另外一个细节:多线程IO主要用在请求完成后返回结果集的过程,也就是socket写操作,至于读socket,单线程本身采用的多路IO复用技术,也不会产生瓶颈,因此Redis给出了一个io-threads-do-reads 参数,决定读socket的时候是否启用多线程。其实io-threads-do-reads是否启用,对性能的影响并不大,最后会做一个验证。
测试环境及策略
本机配置:centos 7,16C+32GB内存Redis版本:6.0.6
翻车
整个测试开始之前,经历了两次翻车才得以继续centos 7上默认的gcc是4.*版本,无法编译Redis 6.0,所以需要升级gcc,因为本机不支持yum安装,参考这里使用源码包安装,gcc编译的时候那个酸爽,本机16C+32GB内存的环境下,因为缺少某些依赖包,导致失败了几次,最终编译成功的一次,花了大概1个小时10分钟
没有认真读配置文件中的说明,设置io-threads后,重启Redis服务后,上来就用redis-benchmark直接怼,其结果跟单线程差不多,令人大跌眼镜。
最后还是在原始配置文件发现了这段话:
If you want to test the Redis speedup using redis-benchmark, make sure you also run the benchmark itself in threaded mode, using the --threads option to match the number of Redis threads, otherwise you'll not be able to notice the improvements.
关于Thread IO的说明
################################ THREADED I/O #################################
# Redis is mostly single threaded, however there are certain threaded
# operations such as UNLINK, slow I/O accesses and other things that are
# performed on side threads.
#
# Now it is also possible to handle Redis clients socket reads and writes
# in different I/O threads. Since especially writing is so slow, normally
# Redis users use pipelining in order to speed up the Redis performances per
# core, and spawn multiple instances in order to scale more. Using I/O
# threads it is possible to easily speedup two times Redis without resorting
# to pipelining nor sharding of the instance.
#
# By default threading is disabled, we suggest enabling it only in machines
# that have at least 4 or more cores, leaving at least one spare core.
# Using more than 8 threads is unlikely to help much. We also recommend using
# threaded I/O only if you actually have performance problems, with Redis
# instances being able to use a quite big percentage of CPU time, otherwise
# there is no point in using this feature.
#
# So for instance if you have a four cores boxes, try to use 2 or 3 I/O
# threads, if you have a 8 cores, try to use 6 threads. In order to
# enable I/O threads use the following configuration directive:
#
# io-threads 4
#
# Setting io-threads to 1 will just use the main thread as usual.
# When I/O threads are enabled, we only use threads for writes, that is
# to thread the write(2) syscall and transfer the client buffers to the
# socket. However it is also possible to enable threading of reads and
# protocol parsing using the following configuration directive, by setting
# it to yes:
#
# io-threads-do-reads no
#
# Usually threading reads doesn't help much.
#
# NOTE 1: This configuration directive cannot be changed at runtime via
# CONFIG SET. Aso this feature currently does not work when SSL is
# enabled.
#
# NOTE 2: If you want to test the Redis speedup using redis-benchmark, make
# sure you also run the benchmark itself in threaded mode, using the
# --threads option to match the number of Redis threads, otherwise you'll not
# be able to notice the improvements.
大多数情况下redis是以单线程的方式运行的,然而,有一些线程操作,如断开链接,耗时的I/O操作(bgsave,expired key清理之类的操作)和其他任务是在side线程(主线程fork出来的子线程)中执行的。
测试结果及分析
从中可以看到:
因此在本机环境下,io-threads 4设置成2或者4个都ok,最多不超过8个,超出后性能反而会下降,同时也不能超出cpu的个数,正如配置文件中注释中说的,至少要留出一个CPU。
关于io-threads-do-reads参数
上文提到过io-threads-do-reads这个参数,它是决定socket读操作是否开启多线程,Redis的socket读操作采用多路IO复用技术,本身不会成为瓶颈,所以这个参数对多线程下测试影响比较小。依旧参考这里的这个图 侵删,这个io-threads-do-reads在笔者理解起来,就是socket读的线程,是否开启影响不大。
Redis.conf中关于此参数的注释为:When I/O threads are enabled, we only use threads for writes, that is to thread the write(2) syscall and transfer the client buffers to the socket. However it is also possible to enable threading of reads and protocol parsing using the following configuration directive, by setting it to yes Usually threading reads doesn't help much.
以下就该参数启用与否进行一次对比测试
参考如下截图,在开启了两个线程的情况下,分别开启和禁用io-threads-do-reads,从整体上看,性能影响差别很小。当然专业的大神可以从源码的角度去分析。
io-threads为2,且启动io-threads-do-reads
io-threads为2,且禁动io-threads-do-reads
多线程版本的Redis和Redis Cluster的选择
redis集群有两种模式:sentinel和cluster,这里暂时先不提sentinel,来思考多线程版本的Redis和Redis Cluster的选择问题。
Redis的Cluster解决的就是扩展性和单节点单线程的阻塞隐患,如果Redis支持了多线程(目前多线程的Redis最对不建议超出8个线程),在不考虑单个节点网卡瓶颈的情况下,其实这两个问题都已经解决了,单节点可以支持多线程和充分利用多核CPU,单节点可以支持到25W QPS,还要啥自行车?
同时要考虑到Redis cluster的痛点:
1,不支持multiple/pipline操作(第三方插件也不一定稳定)。
2,cluster中每个主节点要挂一个从节点,不管这个节点是不是独立的物理节点还是单机多实例中的一个节点,终究是增加了维护成本。
3,只能使用一个数据库
4,集群自身扩容、缩容带来的一系列slot迁移等性能问题,以及集群的管理问题
这些所谓的痛点也不复存在了,所以这里就面临一个重新选择的问题:是使用多线程版本的Redis,还是使用Redis cluster?这是一个需要思考的问题。
疑问
关于redis-benchmark 测试时候 ./bin/redis-benchmark -d 128 -c 200 -n 1000000 -t set -q --threads 2,涉及的两个参数-c和--threads,个人是不太理解的
-c的解释是:指定并发连接数
--threads是redis-benchmark的线程数?
对此去跟老钱(素未谋面,感谢)求证了一下,说该参数是redis-benchmark客户端的epoll,服务器端的多路IO复用原理已经看得我七荤八素了,客户端也是带epoll的,还是不太理解这两者之间的关系。
redis-benchmark测试现场
如下是redis-benchmark测试过程中部分截图