java处理ES查询突破10000条的限制方案

最新推荐文章于 2025-01-05 16:17:53 发布

CatServer

最新推荐文章于 2025-01-05 16:17:53 发布

阅读量2.2k

点赞数 2

CC 4.0 BY-SA版权

分类专栏： Elaticsearch 文章标签： java elasticsearch windows

本文链接：https://blog.csdn.net/CuiGongZi/article/details/133179044

Elaticsearch 专栏收录该内容

2 篇文章

订阅专栏

博客主要围绕Elasticsearch数据限制展开，先介绍了限制10000条数据的情况，接着给出突破该限制的办法，一是使用scroll API，可在不加载全量数据时获取结果；二是修改ES默认设置，但可能导致内存溢出，不推荐。最后说明了具体方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.为什么要限制10000条

 默认情况下，Elasticsearch集群中每个分片的搜索结果数量限制为10000。这是为了避免潜在的性能问题。

2.突破10000条限制的办法

使用scroll API：scroll API可以帮助我们在不加载所有数据的情况下获取所有结果。它会在后台执行查询以获取滚动ID，并将其用于进行后续查询。这样就可以一次性获取所有结果，而不必担心限制
修改ES默认设置,将10000修改成10万或者50万,这样的话要考虑ES内存溢出的问题,不推荐

3.具体方法

使用scroll API

public ConcurrentLinkedDeque<FramePickingInfoVo> getSearchRequest(FramePickingPageVo vo){
        long start = System.currentTimeMillis();
        ConcurrentLinkedDeque<FramePickingInfoVo> list=new ConcurrentLinkedDeque<>();
        try{
            //构建查询请求
            SearchRequest baseSubOrderIndexRequest = CloudBaseQueryBuilder.getBaseSubOrderIndexRequest();
            SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
            BoolQueryBuilder boolQuery = CloudBaseQueryBuilder.getBaseFramePickBoolQuery(vo,true);
            //设置size(10000)
            sourceBuilder.size(batchSize);
            sourceBuilder.query(boolQuery);
            //设置排序方式
            getSort(vo.getOrderByType(),sourceBuilder);
            // 滚动查询请求
            SearchRequest searchRequest = new SearchRequest(BASE_SUB_ORDER_DIMENSION_INDEX);
            searchRequest.source(sourceBuilder);
            //设置请求滚动时间窗口时间
            searchRequest.scroll(TimeValue.timeValueMinutes(scrollTime));

            //执行首次检索
            SearchResponse searchResponse = elasticsearchClient.search(searchRequest, RequestOptions.DEFAULT);

            //首次检索返回scrollId，用于下一次的滚动查询
            String scrollId = searchResponse.getScrollId();
            //获取首次检索命中结果

            SearchHit[] searchHits = searchResponse.getHits().getHits();
            ConcurrentLinkedDeque<FramePickingInfoVo> framPickDateInfos = getFramPickDateInfos(searchResponse);
            list.addAll(framPickDateInfos);
            // 处理滚动结果
            while (searchHits != null && searchHits.length > 0) {
                SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
                scrollRequest.scroll(TimeValue.timeValueMinutes(scrollTime));

                searchResponse = elasticsearchClient.scroll(scrollRequest, RequestOptions.DEFAULT);
                scrollId = searchResponse.getScrollId();
                searchHits = searchResponse.getHits().getHits();
                ConcurrentLinkedDeque<FramePickingInfoVo> framPickDateInfos1 = getFramPickDateInfos(searchResponse);
                list.addAll(framPickDateInfos1);
            }

            // 清理滚动上下文
            ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
            clearScrollRequest.addScrollId(scrollId);

            ClearScrollResponse clearScrollResponse = elasticsearchClient.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
            boolean succeeded = clearScrollResponse.isSucceeded();

            long end = System.currentTimeMillis();
            System.out.println("共执行时间：" + (end - start) / 1000 + " s");

        } catch (Exception e) {
            System.out.println("===error==" + e.getMessage());
            e.printStackTrace();
        }
        return list;
    }

修改ES默认配置

# 调大查询窗口巨细，比方100w
PUT index/_settings
{
  "index.max_result_window": "1000000"
}
# 检查查询最大数
GET index/_settings
---
{
  "demo_scroll" : {
    "settings" : {
      "index" : {
        "number_of_shards" : "5",
        "provided_name" : "demo_scroll",
        "max_result_window" : "1000000",
        "creation_date" : "1680832840425",
        "number_of_replicas" : "1",
        "uuid" : "OLV5W_D9R-WBUaZ_QbGeWA",
        "version" : {
          "created" : "6082399"
        }
      }
    }
  }
}