生产环境elasticsearch5.0报错IllegalArgumentException: number of documents in the index cannot exceed 2147483519的处理

最近几天的push:user:req数据写不到集群，报错如下：

[2017-06-13T09:31:51,803][DEBUG][o.e.a.b.TransportShardBulkAction] [yunva_etl_es8] [push:user:req][1] failed to execute bulk item (index) index {[push:user:req][/push/user/req][AVyfE_JChh2DyKB0G1Kj], source[{"appId":"500062","yunvaId":"362b103623aa4f75911b374615c6a682","providerId":"1","proviceId":"27","mac":"B0:C4:E7:61:E6:E7","imsi":"460023080802034","imei":"354026057092725","factory":"samsung","model":"GT-S7568","osType":"android","osVersion":15,"networkType":"wifi","ip":"14.152.80.239","sessionId":"20170613091139059173449","time":"2017-06-13 09:11:39.061.+0800"}]}
java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519
at org.apache.lucene.index.DocumentsWriterPerThread.reserveOneDoc(DocumentsWriterPerThread.java:208) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:215) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1562) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:558) ~[elasticsearch-5.0.1.jar:5.0.1]

搜索了下：
是lucene的文档数限制，每个分片最大支持2的31次方个文档数量
https://discuss.elastic.co/t/lucene-max-documents-limit/34761
Lucene max documents limit，Yep, there is a 2^31 limit per shard in Lucene.

查询当前es分片文档：

# curl 10.26.241.237:9200/_cat/shards?v
index shard prirep state docs store ip node

push:user:req 1 p STARTED 2147483519 883.7gb 10.30.136.143 yunva_etl_es8
push:user:req 1 r STARTED 2147483519 883.7gb 10.25.135.215 yunva_etl_es2
push:user:req 3 p STARTED 2147483519 883.7gb 10.45.150.115 yunva_etl_es9
push:user:req 3 r STARTED 2147483519 883.7gb 10.30.136.143 yunva_etl_es8
push:user:req 4 r STARTED 2147483519 883.9gb 10.30.136.143 yunva_etl_es8
push:user:req 4 p STARTED 2147483519 883.9gb 10.174.12.230 yunva_etl_es10
push:user:req 2 p STARTED 2147483519 884gb 10.174.12.230 yunva_etl_es10
push:user:req 2 r STARTED 2147483519 884gb 10.27.78.228 yunva_etl_es5
push:user:req 0 r STARTED 2147483519 883gb 10.45.150.115 yunva_etl_es9
push:user:req 0 p STARTED 2147483519 883gb 10.174.12.230 yunva_etl_es10

查询出来的2147483519和2的31次方2147483648接近，所以新增加的数据无法插入

临时的解决办法：先把kafka的磁盘加到数据多保留一段时间20天

原来的集群数据抽取到hbase中作为冷数据处理，再把kafka里面的数据导入数据到新的集群中