Lucene 查询分页技术

常用的Lucene查询代码如下所示，该代码的作用是将path路径下的所有索引信息返回

public String matchAll(String path) {
        try {
            Directory directory = FSDirectory.open(new File(path));
            DirectoryReader reader = DirectoryReader.open(directory);
            IndexSearcher searcher = new IndexSearcher(reader);
            MatchAllDocsQuery query = new MatchAllDocsQuery();
            
            ScoreDoc[] hits = searcher.search(query, null, Integer.MAX_VALUE).scoreDocs;
            StringBuffer buffer = new StringBuffer();
            for (int i = 0; i < hits.length; i++) {
                Document hitDocument = searcher.doc(hits[i].doc);
//                System.out.println(hitDocument.get("key")
//                        + "......"+hitDocument.get("value"));
                buffer.append(hitDocument.get("key")+";"+hitDocument.get("value")+"|");
            }
            return buffer.toString();
        } catch (IOException e) {
            e.printStackTrace();
        }
        return null;
    }

但是当该文件夹下索引的数目比较巨大，那么在执行以下代码的时候，则会出现java.lang.OutOfMemoryError: Java heap space的提示

ScoreDoc[] hits = searcher.search(query, null, Integer.MAX_VALUE).scoreDocs;

这时候，我们可以考虑使用分页技术，比如以前大约1亿条数据，我们可以将其分成100个100W的页，每次对100W条索引数据进行处理，这样就可以避免上述情况的发生。在Lucene 中，我们使用searchAfter的方法实现上述功能。它的官方API介绍如下所示：

public TopDocs searchAfter(ScoreDoc after,
                  Query query,
                  int n)
                    throws IOException

Finds the top n hits for query where all results are after a previous result (after).

By passing the bottom result from a previous page as after, this method can be used for efficient 'deep-paging' across potentially large result sets.

Throws:

BooleanQuery.TooManyClauses

IOException

private String transToContent(IndexSearcher searcher,TopDocs topDocs) throws IOException {
        ScoreDoc[] scoreDocs = topDocs.scoreDocs;
        StringBuffer sb = new StringBuffer();
        for(int i=0; i<scoreDocs.length; i++) {
            Document doc  = searcher.doc(scoreDocs[i].doc);
            sb.append(doc.get("key")+";"+doc.get("value")+"|");
        }
        return sb.toString();
    }
        
    private void matchAll(String path) {
        try {
            Directory directory = FSDirectory.open(new File(path));
            DirectoryReader reader = DirectoryReader.open(directory);
            IndexSearcher searcher = new IndexSearcher(reader);
            
            ScoreDoc after = null;
            TopDocs topDocs = searcher.searchAfter(after, new MatchAllDocsQuery(), Preference.PAGE_SIZE);
            int curPage = 1;
            while(topDocs.scoreDocs.length > 0) {
                System.out.println("Current Page:"+ (curPage++) );
                System.out.println(transToContent(searcher, topDocs));
                after = topDocs.scoreDocs[topDocs.scoreDocs.length -1];
                topDocs = searcher.searchAfter(after, new MatchAllDocsQuery(), Preference.PAGE_SIZE);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }