POI-处理大Excel文件（xls）

最近需要处理一个比较大的excel文件，但是poi在处理文件时会抛出OOM导致程序崩溃，查看官方文档看到可以以流式的方式读取excel避免读取大文件时的OOM。本文主要记述xls的处理。

环境模拟

先准备一个大的excel文件（xls大小8M），再将jvm的heap缩小到100m(JVM 参数 -Xmx100m)用于模拟OOM
并使用参数在OOM时dump内存 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=d://dump.hprof

使用HSSF读取

    public static void main(String [] args) throws IOException {
        InputStream is = new FileInputStream("d://large.xls");
        Workbook wb = new HSSFWorkbook(is);
    }

运行之后

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.lang.Integer.valueOf(Integer.java:832)
	at org.apache.poi.hssf.usermodel.HSSFSheet.getRow(HSSFSheet.java:385)
	at org.apache.poi.hssf.usermodel.HSSFSheet.setPropertiesFromSheet(HSSFSheet.java:212)
	at org.apache.poi.hssf.usermodel.HSSFSheet.<init>(HSSFSheet.java:145)
	at org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:353)
	at org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:398)
	at org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:379)
	at blog.excel.EventExample.main(EventExample.java:109)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

就报出OOM，虽然文件只有8M并且heap有100m但是还是发生了OOM，使用jvisualvm查看dump的内容

可以看到创建了许多poi内部的Record（由于我excel内都是数字所以都是NumberRecord）。

使用SAX流式读取

POI提供了SAX流式读取的方式，可以减小内存的使用

//需要继承HSSFListener
public class EventExample  implements HSSFListener
{
    private SSTRecord sstrec;

    /**
     * 实现接口方法用于处理每一条记录，包括workbook/row/cell
     */
    @Override
    public void processRecord(Record record)
    {
        switch (record.getSid())
        {
            // 标记workbook或sheet开始，这里会进行判断
            case BOFRecord.sid:
                BOFRecord bof = (BOFRecord) record;
                if (bof.getType() == bof.TYPE_WORKBOOK)
                {
                    System.out.println("Encountered workbook");
                    // assigned to the class level member
                } else if (bof.getType() == bof.TYPE_WORKSHEET)
                {
                    System.out.println("Encountered sheet reference");
                }
                break;
            //处理sheet
            case BoundSheetRecord.sid:
                BoundSheetRecord bsr = (BoundSheetRecord) record;
                System.out.println("New sheet named: " + bsr.getSheetname());
                break;
            //处理行
            case RowRecord.sid:
                RowRecord rowrec = (RowRecord) record;
                System.out.println("Row found, first column at "
                        + rowrec.getFirstCol() + " last column at " + rowrec.getLastCol());
                break;
            //处理数字单元格
            case NumberRecord.sid:
                NumberRecord numrec = (NumberRecord) record;
                System.out.println("Cell found with value " + numrec.getValue()
                        + " at row " + numrec.getRow() + " and column " + numrec.getColumn());
                break;
            // 包含一行中所有文本单元格
            case SSTRecord.sid:
                sstrec = (SSTRecord) record;
                for (int k = 0; k < sstrec.getNumUniqueStrings(); k++)
                {
                    System.out.println("String table value " + k + " = " + sstrec.getString(k));
                }
                break;
            //处理文本单元格
            case LabelSSTRecord.sid:
                LabelSSTRecord lrec = (LabelSSTRecord) record;
                System.out.println("String cell found with value "
                        + sstrec.getString(lrec.getSSTIndex()));
                break;
        }
    }

    public static void main(String[] args) throws IOException
    {
        // 文件inputStream
        FileInputStream is = new FileInputStream("d://large.xls");
        POIFSFileSystem poifs = new POIFSFileSystem(is);
        InputStream din = poifs.createDocumentInputStream("Workbook");
        HSSFRequest req = new HSSFRequest();
        // 为HSSFRequest增加listener
        req.addListenerForAllRecords(new EventExample());
        HSSFEventFactory factory = new HSSFEventFactory();
        // 处理inputstream
        factory.processEvents(req, din);
        // 关闭inputstream
        is.close();
        din.close();
        System.out.println("done.");
    }
}

不足

使用以上的方法可以读取大文件，而不用担心oom发生。但是只能读到其单元格字面值，HSSF中的样式等都无法获取，而且只能进行读取无法修改、添加，这大大限制了其使用的范围。如果需要进行大文件的追加就需要后面介绍的SXSSF进行处理了。