为何要自行编译hadoop源码,往往是由于官方提供的hadoop发行版都是基于32位操作系统,在操作hadoop时会发生warn。
准备软件:
1)JDK
2)Hadoop源码
3)Maven
4)Protobuf
编译小结:
1)JDK安装(略)
2)必要依赖安装
sudo yum install -y g++ gcc gcc-c++ autoconf automake libtoolcmake zliblg-dev pkg-conf libssl-dev openssh-client make mcurses-devel
3)Protobuf安装
./configure --prefix=/usr/local/protoc/
make
make install
4)Maven安装(略)
5)vi .bash_profile
export MAVEN_HOME=/opt/softwares/apache-maven-3.2.5
PATH=$MAVEN_HOME/bin:/usr/local/protoc/bin:$PATH
6)Hadoop编译
mvn package –DskipTests –Pdist,native
7)编译成功后,将hadoop-dist/target/hadoop-2.3.0-cdh5.0.2/lib/native下文件拷贝纸$HADOOP_HOME/lib/native下,文件如下:
libhadoop.a
libhadooppipes.a
libhadoop.so -> libhadoop.so.1.0.0
libhadoop.so.1.0.0
libhadooputils.a
libhdfs.a
libhdfs.so -> libhdfs.so.0.0.0
libhdfs.so.0.0.0
8)验证
hadoop checknative
15/11/06 15:46:41 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
15/11/06 15:46:41 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /opt/modules/hadoop-2.3.0-cdh5.0.2/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: false
lz4: true revision:99
bzip2: false