zoukankan      html  css  js  c++  java
  • Java代码解决ElasticSearch的Result window is too large问题

    调用ElasticSearch做分页查询时报错:

    QueryPhaseExecutionException[Result window is too large, from + size must be less than or equal to: [10000] but was [666000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.]; }

    提示用from+size方式有1万条数据查询的限制,需要更改index.max_result_window参数的值。

    翻了下elasticsearch官网的文档:

    index.max_result_window
    The maximum value of from + size for searches to this index.Defaults to 10000. 
    Search requests take heap memory and time proportional to from + size and this limits that memory.
    See Scroll or Search After for a more efficient alternative to raising this.

    说是用传统方式(from + size)查询占用内存空间且比较消耗时间,所以做了限制。

    问题是用scroll方式做后台分页根本行不通。

    不说用scroll方式只能一页页的翻这种不人性化的操作。页码一多,scrollId也很难管理啊。

    所以继续鼓捣传统方式的分页。

    上网查了下设置max_result_window的方法,全都是用crul或者http方式改的。

    后来无意间看到了一篇文档: https://blog.csdn.net/tzconn/article/details/83309516

    结合之前逛elastic中文社区的时候知道这个参数是索引级别的。于是小试了一下,结果竟然可以了。

    java代码如下:

    public SearchResponse search(String logIndex, String logType, QueryBuilder query, 
    List<AggregationBuilder> agg, int page, int size) { page = page > 0 ? page - 1 : page; TransportClient client = getClient(); SearchRequestBuilder searchRequestBuilder = client.prepareSearch(logIndex.split(",")) .setTypes(logType.split(",")) .setSearchType(SearchType.DFS_QUERY_THEN_FETCH) .addSort("createTime", SortOrder.DESC); if (agg != null && !agg.isEmpty()) { for (int i = 0; i < agg.size(); i++) { searchRequestBuilder.addAggregation(agg.get(i)); } } updateIndexs(client, logIndex, page, size); SearchResponse searchResponse = searchRequestBuilder .setQuery(query) .setFrom(page * size) .setSize(size) .get(); return searchResponse; } //更新索引的max_result_window参数 private boolean updateIndexs(TransportClient client, String indices, int from, int size) { int records = from * size + size; if (records <= 10000) return true; UpdateSettingsResponse indexResponse = client.admin().indices() .prepareUpdateSettings(indices) .setSettings(Settings.builder() .put("index.max_result_window", records) .build() ).get(); return indexResponse.isAcknowledged(); }

    搞定。

    当然这段代码不好的地方在于:

    每次查询超过10000万条记录的时候,都会去更新一次index。

    这对原本就偏慢的from+size查询来说,更是雪上加霜了。

  • 相关阅读:
    自动从vss下载代码并编译的脚本
    自动执行Sql脚本的批处理
    编译 SharpDevelop2.2源码 经过
    .net框架源代码批量下载器使用方法
    sql server 解决孤立用户问题
    元数据学习分享
    SQL注入攻击的原理及其防范措施(转)
    .Net Remoting和Web Service浅析(转)
    usercontrol 和 page 的继承
    如何提高页面现实速度
  • 原文地址:https://www.cnblogs.com/braska/p/10007812.html
Copyright © 2011-2022 走看看