zoukankan      html  css  js  c++  java
  • 生产环境elasticsearch5.0报错IllegalArgumentException: number of documents in the index cannot exceed 2147483519的处理

    最近几天的push:user:req数据写不到集群,报错如下:

    [2017-06-13T09:31:51,803][DEBUG][o.e.a.b.TransportShardBulkAction] [yunva_etl_es8] [push:user:req][1] failed to execute bulk item (index) index {[push:user:req][/push/user/req][AVyfE_JChh2DyKB0G1Kj], source[{"appId":"500062","yunvaId":"362b103623aa4f75911b374615c6a682","providerId":"1","proviceId":"27","mac":"B0:C4:E7:61:E6:E7","imsi":"460023080802034","imei":"354026057092725","factory":"samsung","model":"GT-S7568","osType":"android","osVersion":15,"networkType":"wifi","ip":"14.152.80.239","sessionId":"20170613091139059173449","time":"2017-06-13 09:11:39.061.+0800"}]}
    java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519
    at org.apache.lucene.index.DocumentsWriterPerThread.reserveOneDoc(DocumentsWriterPerThread.java:208) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
    at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:215) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
    at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
    at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1562) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
    at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
    at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:558) ~[elasticsearch-5.0.1.jar:5.0.1]

    搜索了下:
    是lucene的文档数限制,每个分片最大支持2的31次方个文档数量
    https://discuss.elastic.co/t/lucene-max-documents-limit/34761
    Lucene max documents limit,Yep, there is a 2^31 limit per shard in Lucene.

    查询当前es分片文档:

    # curl 10.26.241.237:9200/_cat/shards?v
    index shard prirep state docs store ip node
    
    push:user:req 1 p STARTED 2147483519 883.7gb 10.30.136.143 yunva_etl_es8
    push:user:req 1 r STARTED 2147483519 883.7gb 10.25.135.215 yunva_etl_es2
    push:user:req 3 p STARTED 2147483519 883.7gb 10.45.150.115 yunva_etl_es9
    push:user:req 3 r STARTED 2147483519 883.7gb 10.30.136.143 yunva_etl_es8
    push:user:req 4 r STARTED 2147483519 883.9gb 10.30.136.143 yunva_etl_es8
    push:user:req 4 p STARTED 2147483519 883.9gb 10.174.12.230 yunva_etl_es10
    push:user:req 2 p STARTED 2147483519 884gb 10.174.12.230 yunva_etl_es10
    push:user:req 2 r STARTED 2147483519 884gb 10.27.78.228 yunva_etl_es5
    push:user:req 0 r STARTED 2147483519 883gb 10.45.150.115 yunva_etl_es9
    push:user:req 0 p STARTED 2147483519 883gb 10.174.12.230 yunva_etl_es10

    查询出来的2147483519和2的31次方2147483648接近,所以新增加的数据无法插入

    临时的解决办法:先把kafka的磁盘加到数据多保留一段时间20天

    原来的集群数据抽取到hbase中作为冷数据处理,再把kafka里面的数据导入数据到新的集群中

  • 相关阅读:
    POJ 3672 水题......
    POJ 3279 枚举?
    STL
    241. Different Ways to Add Parentheses
    282. Expression Add Operators
    169. Majority Element
    Weekly Contest 121
    927. Three Equal Parts
    910. Smallest Range II
    921. Minimum Add to Make Parentheses Valid
  • 原文地址:https://www.cnblogs.com/reblue520/p/7009305.html
Copyright © 2011-2022 走看看