Elasticsearch GC 时间过长的解决方法

前言：GC 时间过长是个常见的问题，下文我将对应的现象和解决方案进行阐述。为什么这么解决，可以参考我的另外一个博客中的内存使用和GC指标这个章节

我们有时会发现elasticsearch集群挂掉，或者有点数据节点脱离集群，这里有可能是GC方面的原因，实质是内存的原因。

一、日志表现

[2017-06-22 23:56:51,008][WARN ][monitor.jvm              ] [data-vm0] [gc][old][5214195][124260] duration [22.4s], collections [1]/[23s], total [22.4s]/[4.2d], memory [13gb]->[13gb]/[13.6gb], all_pools {[young] [21.7mb]->[25.2mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [13gb]->[13gb]/[13gb]}
[2017-06-22 23:57:21,419][WARN ][monitor.jvm              ] [data-vm0] [gc][old][5214196][124261] duration [29.6s], collections [1]/[30.4s], total [29.6s]/[4.2d], memory [13gb]->[13gb]/[13.6gb], all_pools {[young] [25.2mb]->[32.7mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [13gb]->[13gb]/[13gb]}
[2017-06-22 23:57:43,963][WARN ][monitor.jvm              ] [data-vm0] [gc][old][5214197][124262] duration [22.3s], collections [1]/[22.5s], total [22.3s]/[4.2d], memory [13gb]->[12.9gb]/[13.6gb], all_pools {[young] [32.7mb]->[19.4mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [13gb]->[12.9gb]/[13gb]}
[2017-06-22 23:58:14,390][WARN ][monitor.jvm              ] [data-vm0] [gc][old][5214198][124263] duration [30.1s], collections [1]/[30.4s], total [30.1s]/[4.2d], memory [12.9gb]->[13gb]/[13.6gb], all_pools {[young] [19.4mb]->[28.5mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [12.9gb]->[13gb]/[13gb]}
[2017-06-22 23:58:37,356][WARN ][monitor.jvm              ] [data-vm0] [gc][old][5214199][124264] duration [22.5s], collections [1]/[22.9s], total [22.5s]/[4.2d], memory [13gb]->[13gb]/[13.6gb], all_pools {[young] [28.5mb]->[9.9mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [13gb]->[13gb]/[13gb]}
[2017-06-22 23:59:07,774][WARN ][monitor.jvm              ] [data-vm0] [gc][old][5214200][124265] duration [29.9s], collections [1]/[30.4s], total [29.9s]/[4.2d], memory [13gb]->[13gb]/[13.6gb], all_pools {[young] [9.9mb]->[1.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [13gb]->[13gb]/[13gb]}
[2017-06-22 23:59:40,430][WARN ][monitor.jvm              ] [data-vm0] [gc][old][5214205][124266] duration [27.6s], collections [1]/[28.6s], total [27.6s]/[4.2d], memory [13.5gb]->[13gb]/[13.6gb], all_pools {[young] [515.7mb]->[73.7mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [13gb]->[13gb]/[13gb]}

二、解决办法

出现上述情况的日志，说明节点正在承受内存方面的压力。

1.增加节点的内存,纵向分担压力（%50的RAM 并不大于32G）
2.增加节点数量，横向分担压力

NOTE: 可能有些小伙伴会清理caches来减轻压力。我认为此处清理cache是不管用的，因为下次查询的时候 fielddata 会被reload。