RocksDB Rate Limiter源码解析

这次的项目我们重点关注RocksDB中的一个环节：Rate Limiter。其实Rate Limiter的思想在很多其他系统中也很常用。

在RocksDB中，后台会实时运行compaction和flush操作，这些都会对磁盘进行大量的写操作。可以通过Rate Limiter来控制最大写入速度的上限。因为在某些场景下，突发的大量写入会导致很大的read latency，从而影响系统性能。

Rate Limiter的基本原理是令牌桶算法：系统每秒往桶里丢数量为1/QPS的令牌（满了为止），写请求只有拿到了令牌才能处理。当桶里没有令牌时便可拒绝服务（阻塞）。它在RocksDB中的实现可以参考这里。

Rate Limiter中可以调节的有以下几个参数：

• int64_t rate_bytes_per_sec：控制 compaction 和 flush 每秒总写入量的上限。一般情况下只需要调节这一个参数。
• int64_t refill_period_us：控制 tokens 多久再次填满，譬如 rate_limit_bytes_per_sec 是 10MB/s，而 refill_period_us 是 100ms，那么每 100ms 的流量就是 1MB/s。
• int32_t fairness：用来控制 high 和 low priority 的请求，防止 low priority 的请求饿死。

更详细的介绍可以直接看rate_limiter.h：

// Create a RateLimiter object, which can be shared among RocksDB instances to control write rate of flush and compaction.
// @rate_bytes_per_sec: this is the only parameter you want to set most of the time. It controls the total write rate of compaction and flush in bytes per second. Currently, RocksDB does not enforce rate limit for anything other than flush and compaction, e.g. write to WAL.
// @refill_period_us: this controls how often tokens are refilled. For example, when rate_bytes_per_sec is set to 10MB/s and refill_period_us is set to 100ms, then 1MB is refilled every 100ms internally. Larger value can lead to burstier writes while smaller value introduces more CPU overhead. The default should work for most cases.
// @fairness: RateLimiter accepts high-pri requests and low-pri requests.  A low-pri request is usually blocked in favor of hi-pri request. Currently, RocksDB assigns low-pri to request from compaction and high-pri to request from flush. Low-pri requests can get blocked if flush requests come in continuously. This fairness parameter grants low-pri requests permission by 1/fairness chance even though high-pri requests exist to avoid starvation. You should be good by leaving it at default 10.
// @mode: Mode indicates which types of operations count against the limit.
// @auto_tuned: Enables dynamic adjustment of rate limit within the range `[rate_bytes_per_sec / 20, rate_bytes_per_sec]`, according to the recent demand for background I/O.
extern RateLimiter* NewGenericRateLimiter(
   int64_t rate_bytes_per_sec, int64_t refill_period_us = 100 * 1000,
   int32_t fairness = 10,
   RateLimiter::Mode mode = RateLimiter::Mode::kWritesOnly,
   bool auto_tuned = false);

}  // namespace rocksdb

这里有个bool auto_tuned，这是RocksDB中带的一个Auto tune Rate Limiter的模块。因为这个rate_bytes_per_sec（写入速度的上限）是很难手动调节的，太大了没效果，太小了又会导致大量写操作无法继续。所以RocksDB提供了这个模块来自动调节。当开启这个模块时，参数rate_bytes_per_sec的含义就变成了定义写入速度上限的上限（In this case rate_bytes_per_sec will indicate the upper-bound of the window within which a rate limit will be picked dynamically.）。之后这个auto tuner会周期性监测I/O写入量，并相应增大/减小写入量上限的值（重新设置rate_bytes_per_sec_和refill_bytes_per_period_）。这里的benchmark显示Auto-tune Rate Limiter可以有效减少write突然增加的程度。

---

Rate Limiter的用法

可以通过NewGenericRateLimiter类来新建一个Rate Limiter。可以对每个RocksDB实例单独搞一个，也可以让所有实例共享一个来control the aggregated write rate of flush and compaction。

RateLimiter* rate_limiter = NewGenericRateLimiter(
    rate_bytes_per_sec /* int64_t */, 
    refill_period_us /* int64_t */,
    fairness /* int32_t */);

这里面三个参数的含义上面介绍过啦。

Although tokens are refilled with a certain interval set by refill_period_us, the maximum bytes that can be granted in a single burst have to be bounded since we are not happy to see that tokens are accumulated for a long time and then consumed by a single burst request which definitely does not agree with our intention. GetSingleBurstBytes() returns this upper bound of tokens.

使用时，在每次写请求之前，都要申请token。如果当前请求无法满足（也就是被限速了），请求就会被阻塞，直到token被填充到足够完成请求。

// block if tokens are not enough
rate_limiter->Request(1024 /* bytes */, rocksdb::Env::IO_HIGH); 

// perform a write operation
Status s = db->Flush();

在运行过程中，也可以通过 SetBytesPerSecond(int64_t bytes_per_second) 动态修改Rate Limiter的流量。

Ref：

https://rocksdb.org.cn/doc/Rate-Limiter.html

https://github.com/facebook/rocksdb/wiki/Rate-Limiter

https://www.cnblogs.com/cchust/p/6007486.html

https://github.com/facebook/rocksdb/wiki/Statistics

RocksDB Compaction：

https://rimzy.net/category/graphs/

http://yinqiwen.github.io/Ardb/2014/09/13/ardb-practice.html

https://www.reddit.com/r/IAmA/comments/3de3cv/we_are_rocksdb_engineering_team_ask_us_anything/

 https://www.cnblogs.com/pdev/p/11277784.html

 http://mysql.taobao.org/monthly/2018/11/05/

 http://mysql.taobao.org/monthly/2018/12/08/

https://github.com/facebook/rocksdb/wiki/Basic-Operations

..