Hadoop 文件压缩

1、指定编解码器类

  public void deflateCompress() throws IOException {
        //获取deflate编解码器类
        Class codecClass = DeflateCodec.class;
        //实例化
        CompressionCodec codec =  (CompressionCodec)ReflectionUtils.newInstance(codecClass,new Configuration());
        //创建文件输出流
        FileOutputStream fos =  new FileOutputStream("D:/compress/1.deflate");
        //得到文件压缩流
        CompressionOutputStream zipout = codec.createOutputStream(fos);
        IOUtils.copyBytes(new FileInputStream("D:/compress/hello.txt"),zipout,1024);
        zipout.close();
    }

2、不指定编解码器类-通用压缩测试

 @Test
    public void deflateCompress() throws IOException {
      Class[] zipclass = {
              DeflateCodec.class,
              GzipCodec.class,
              BZip2Codec.class,
      };
      for(Class c : zipclass){
          commonCompress(c);
      }
    }

    public void commonCompress(Class codecClass) throws IOException {
        //获取当前系统毫秒数
        long start = System.currentTimeMillis();
        //实例化
        CompressionCodec codec =  (CompressionCodec)ReflectionUtils.newInstance(codecClass,new Configuration());
        //创建文件输出流,得到默认扩展流
        FileOutputStream fos =  new FileOutputStream("D:/compress/compress" + codec.getDefaultExtension());
        //得到文件压缩流
        CompressionOutputStream zipout = codec.createOutputStream(fos);
        IOUtils.copyBytes(new FileInputStream("D:/compress/hello.txt"),zipout,1024);
        zipout.close();
        System.out.println(codecClass.getSimpleName() + " : " + (System.currentTimeMillis()-start)+ "ms" );

    }

测试结果

DeflateCodec : 441ms
GzipCodec : 4ms
BZip2Codec : 55ms

3、解压缩

//解压
    public void unzip(Class codecClass) throws IOException {
        //获取当前系统毫秒数
        long start = System.currentTimeMillis();
        //实例化
        CompressionCodec codec =  (CompressionCodec)ReflectionUtils.newInstance(codecClass,new Configuration());
        //创建文件输入流,
        FileInputStream fis  =  new FileInputStream("D:/compress/compress" + codec.getDefaultExtension());
        //得到文件压缩流
        CompressionInputStream zipIn = codec.createInputStream(fis);
        IOUtils.copyBytes(zipIn,new FileOutputStream("D:/compress/unzip" + ".txt"),1024);
        zipIn.close();
        System.out.println(codecClass.getSimpleName() + " : " + (System.currentTimeMillis()-start) + "ms");

    }

DeflateCodec : 414ms
GzipCodec : 44ms
Disconnected from the target VM, address: '127.0.0.1:61073', transport: 'socket'
BZip2Codec : 45ms