配置hadoop压缩 : hadoop build lzo

1.下载lzo

wget 'https://github.com/cloudera/hadoop-lzo/tarball/0.4.14' -O hadoop-lzo-0.4.14.tar.gz --no-check-certificate

2.编译lzo

tar zxf  hadoop-lzo-0.4.14.tar.gz

cd ./cloudera-hadoop-lzo-8aa0605/

JAVA_HOME=${JAVA_HOME}   C_INCLUDE_PATH=${HADOOP_HOME}/lzo-2.06/include  LIBRARY_PATH=${HADOOP_HOME}/lzo-2.06/lib

ant compile-native

ant jar

cp -af /usr/local/hadoop-0.20.2-cdh3u3/cloudera-hadoop-lzo-8aa0605/build/hadoop-lzo-0.4.14.jar  /usr/local/hadoop/lib

3.配置lzo

core-site.xml

<!-- compression config -->
        <property>
                <name>io.compression.codecs</name>
                <value>com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec</value>
                <description>A list of the compression codec classes that can be used for compression/decompression.</description>
        </property>

4.配置hadoop Snappy

mapred-site.xml

<!--compression-->
        <property>
          <name>mapred.compress.map.output</name>
          <value>true</value>
        </property>

        <property>
          <name>mapred.map.output.compression.codec</name>
          <value>org.apache.hadoop.io.compress.SnappyCodec</value>
        </property>

        <property>
          <name>mapred.output.compression.type</name>
          <value>BLOCK</value>
        </property>

5.重启hadoop

stop-all.sh

start-all.sh

datanode节点报错：

原因是主namenode重新编译，导致主从节点版本异同。

(以后最好不要在主节点重新编译，这样同步hadoop builde version工作量挺大)

2012-06-05 13:47:23,006 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = datanode113.hadoop/172.16.51.113
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2-cdh3u3
STARTUP_MSG: build = file:///usr/local/hadoop -r Unknown; compiled by 'root' on 2012氓鹿麓 04忙?? 19忙?楼 忙??忙??氓?? 12:48:49 CST
************************************************************/
2012-06-05 13:47:23,260 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
2012-06-05 13:47:23,315 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Incompatible build versions: namenode BV = 318bc781117fa276ae81a3d111f5eeba0020634f; datanode BV = Unknown
2012-06-05 13:47:23,421 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible build versions: namenode BV = 318bc781117fa276ae81a3d111f5eeba0020634f; datanode BV = Unknown
    at org.apache.hadoop.hdfs.server.datanode.DataNode.handshake(DataNode.java:608)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:387)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:305)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1627)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1567)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1585)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1711)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1728)

2012-06-05 13:47:23,422 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at datanode113.hadoop/172.16.51.113
************************************************************/

fix:

备份从节点hadoop 当前版本，将主节点新版本重新scp至各个从节点。

参考：

https://www.opensciencegrid.org/bin/view/Documentation/Release3/HadoopDebug