kibana 报错 FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed

环境

kibana6.4.3

logstash6.4.3

es6.4.3

日志收集服务器内存8g

前景

elk 服务器搭建好的前几天服务都正常

大概一周以后出现kibana 连接不上，登录服务器发现只有 kibana 挂掉，es 和 logstash 都正常运行，排查日志发现：

<--- Last few GCs --->
 
[7232:0x28c9b10] 1349425042 ms: Mark-sweep 1360.5 (1431.3) -> 1360.5 (1431.3) MB, 1475.4 / 0.0 ms  allocation failure GC in old space requested
[7232:0x28c9b10] 1349426671 ms: Mark-sweep 1360.5 (1431.3) -> 1360.5 (1431.3) MB, 1628.7 / 0.1 ms  last resort GC in old space requested
[7232:0x28c9b10] 1349428162 ms: Mark-sweep 1360.5 (1431.3) -> 1360.5 (1431.3) MB, 1490.7 / 0.0 ms  last resort GC in old space requested
<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2e54d3625879 <JSObject>
    2: /* anonymous */(aka /* anonymous */) [/opt/kibana-6.4.3/src/server/status/lib/cgroup.js:70] [bytecode=0x1d79566d9321 offset=20](this=0x754e6a822d1 <undefined>,controller=0x31e957539041 <String[4]: pids>)
    3: arguments adaptor frame: 3->1
    4: forEach(this=0x31e957539061 <JSArray[1]>)
    5: /* anonymous */(aka /* anonymous */) [/opt/kibana-6.4.3/src/server/status/lib/cgroup.js:69] [b...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [./../node/bin/node]
 2: 0x8c21ec [./../node/bin/node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [./../node/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [./../node/bin/node]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [./../node/bin/node]
 6: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::NewInternal(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [./../node/bin/node]
 7: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag, v8::internal::MinimumCapacity) [./../node/bin/node]
 8: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::EnsureCapacity(v8::internal::Handle<v8::internal::StringTable>, int, v8::internal::PretenureFlag) [./../node/bin/node]
 9: v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>) [./../node/bin/node]
10: v8::internal::LookupIterator::PropertyOrElement(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, bool*, v8::internal::LookupIterator::Configuration) [./../node/bin/node]
11: v8::internal::Runtime_SetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [./../node/bin/node]
12: 0x22a4d09842fd

发现是前端 js 报的内存 oom 异常，通过网上资料发现node.js 的默认内存大小为1.4g

Node 中通过 JavaScript 使用内存时只能使用部分内存（64位系统下约为1.4 GB，32位系统下约为0.7 GB），这就是我们编译项目时为什么会出现内存泄露了，因为前端项目如果非常的庞大，webpack 编译时就会占用很多的系统资源，如果超出了V8对 Node 默认的内存限制大小就会出现node内存溢出的报错:CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

进入 kibana/bin 目录（也可以设置在 node 的环境变量里，我是加在 kibana 文件中）

vi kibana

添加内存空间的环境变量（以下 size 为1g，根据实际情况修改）

NODE_OPTIONS="--max-old-space-size=12288"

https://segmentfault.com/a/1190000017972349

https://banyudu.com/posts/dynamic-change-nodejs-memory-limit.3c34c2

https://forum.search-guard.com/t/kibana-7-6-memory-issue-with-sg/1814