Hadoop文件压缩放到远程centos调试

文件压缩

1、准备代码

package com.cr.compress;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.io.compress.*;
import org.apache.hadoop.util.ReflectionUtils;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;

public class compressTest {
    public static void main(String[] args) throws IOException {
        Class[] zipclass = {
                DeflateCodec.class,
                GzipCodec.class,
                BZip2Codec.class,
        };
        for(Class c : zipclass){
            commonCompress(c);
        }
    }
    
    public static void commonCompress(Class codecClass) throws IOException {
        //获取当前系统毫秒数
        long start = System.currentTimeMillis();
        //实例化
        CompressionCodec codec =  (CompressionCodec) ReflectionUtils.newInstance(codecClass,new Configuration());
        //创建文件输出流,
        FileOutputStream fos =  new FileOutputStream("/home/xiaoqiu/zip" + codec.getDefaultExtension());
        //得到文件压缩流
        CompressionOutputStream zipout = codec.createOutputStream(fos);
        IOUtils.copyBytes(new FileInputStream("/home/xiaoqiu/compress/hello.txt"),zipout,1024);
        zipout.close();
        System.out.println(codecClass.getSimpleName() + " : " + (System.currentTimeMillis()-start) + "ms");

    }

    public static void unzip(Class codecClass) throws IOException {
        //获取当前系统毫秒数
        long start = System.currentTimeMillis();
        //实例化
        CompressionCodec codec =  (CompressionCodec) ReflectionUtils.newInstance(codecClass,new Configuration());
        //创建文件输入流,
        FileInputStream fis  =  new FileInputStream("/home/xiaoqiu/zip" + codec.getDefaultExtension());
        //得到文件压缩流
        CompressionInputStream zipIn = codec.createInputStream(fis);
        IOUtils.copyBytes(zipIn,new FileOutputStream("/home/xiaoqiu/compress/unzip" + codec.getDefaultExtension() + ".txt"),1024);
        zipIn.close();
        System.out.println(codecClass.getSimpleName() + " : " + (System.currentTimeMillis()-start) + "ms");
    }

}

2、生成jar包

3、远程调试

[xiaoqiu@s150 /home/xiaoqiu]$ hadoop jar compress.jar com.cr.compress.compressTest

报错：

Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
        at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:284)
        at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:238)
        at java.util.jar.JarVerifier.processEntry(JarVerifier.java:316)
        at java.util.jar.JarVerifier.update(JarVerifier.java:228)
        at java.util.jar.JarFile.initializeVerifier(JarFile.java:383)
        at java.util.jar.JarFile.getInputStream(JarFile.java:450)
        at org.apache.hadoop.util.RunJar.unJar(RunJar.java:101)
        at org.apache.hadoop.util.RunJar.unJar(RunJar.java:81)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:209)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

解决：

删除MET-INF下面的.SF文件

4、调试成功

[xiaoqiu@s150 /home/xiaoqiu]$ hadoop jar compress.jar com.cr.compress.compressTest
17/12/30 14:05:07 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
17/12/30 14:05:08 INFO compress.CodecPool: Got brand-new compressor [.deflate]
DeflateCodec : 2470ms
17/12/30 14:05:08 INFO compress.CodecPool: Got brand-new compressor [.gz]
GzipCodec : 62ms
17/12/30 14:05:08 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
17/12/30 14:05:08 INFO compress.CodecPool: Got brand-new compressor [.bz2]
BZip2Codec : 145ms

文件解压缩

1、准备代码

改变main函数

public static void main(String[] args) throws IOException {
        Class[] zipclass = {
                DeflateCodec.class,
                GzipCodec.class,
                BZip2Codec.class,
        };
        for(Class c : zipclass){
//            commonCompress(c);
            unzip(c);
        }
    }

2、重新生成jar包

这里生成artifact的时候，选择all modules，否则找不到类，因为我的类是在Java模块里面的

查看类路径是否存在类

3、远程调试

[xiaoqiu@s150 /home/xiaoqiu]$ hadoop jar compress.jar com.cr.compress.compressTest
17/12/30 15:12:25 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
17/12/30 15:12:26 INFO compress.CodecPool: Got brand-new decompressor [.deflate]
DeflateCodec : 2666ms
17/12/30 15:12:26 INFO compress.CodecPool: Got brand-new decompressor [.gz]
GzipCodec : 78ms
17/12/30 15:12:26 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
17/12/30 15:12:26 INFO compress.CodecPool: Got brand-new decompressor [.bz2]
BZip2Codec : 97ms

[xiaoqiu@s150 /home/xiaoqiu/compress]$ ll
total 16
-rw-rw-r--. 1 xiaoqiu xiaoqiu 18 Dec 30 09:51 hello.txt
-rw-rw-r--. 1 xiaoqiu xiaoqiu 18 Dec 30 15:12 unzip.bz2.txt
-rw-rw-r--. 1 xiaoqiu xiaoqiu 18 Dec 30 15:12 unzip.deflate.txt
-rw-rw-r--. 1 xiaoqiu xiaoqiu 18 Dec 30 15:12 unzip.gz.txt
[xiaoqiu@s150 /home/xiaoqiu/compress]$