MQ队列堆积太长，消费不过来怎么办(转)

转自：http://windwrite.com/archives/603

我们现有的业务就面临此问题，消息生产太快，消费不过来，导致队列堆积很长，把服务器内存耗尽，这时RabbitMQ的处理能力很低下。

我在RabbitMQ社区发邮件问了下：

We have the queue publisher and subscriber, the subscriber is going with multi-threads.

The problem is the queue has been increasing too quickly, the consumers can not follow the increment speed, as you see:

root@ubuntu:~# for i in `seq 1 10`;do rabbitmqctl list_queues -p /actionlog;sleep 1;done
Listing queues …
queue_storm_actionlog 12665787
…done.
Listing queues …
queue_storm_actionlog 12670873
…done.
Listing queues …
queue_storm_actionlog 12667975
…done.
Listing queues …
queue_storm_actionlog 12672645
…done.
After one day or about, the queue size can be more than 20 millions, thus RabbitMQ becomes high load and the consuming process is very slow.

收到的答复邮件内容如下：

Increase # of throughput of consumers or reduce publishing rate. If consumers cannot
catch up, hardware upgrades will only offset the moment when systems starts running
low on resources.

If messages can be lost, 3.1 introduces a new feature, queue length limit:
http://www.rabbitmq.com/maxlength.html

Further to the suggestion of imposing a maximum queue length, you could
also consider imposing a maximum message age and instruct the broker to
discard messages that become stale: http://www.rabbitmq.com/ttl.html

总结起来解决方案大体包括：

• 增加消费者的处理能力，或减少发布频率
• 单纯升级硬件不是办法，只能起到一时的作用
• 考虑使用队列最大长度限制，RabbitMQ 3.1支持
• 给消息设置年龄，超时就丢弃